There's no inherent function in Thickbox to check, but looking at the code, you should just be able to check for the presence and visibility of #TB_window
:
var thickbox_shown = ($('#TB_window').is(':visible')) ? true : false;
Or, try counting the length of #TB_load
:
var thickbox_shown = ($('#TB_load').length > 0) ? true : false;\