Found something that may help you:
nsIWindowMediator is a component that keeps track of open windows. It has getEnumerator
method that allows you to get all open windows.
If you loop through it, and count them, you get the number of open windows, like so:
var windowMediator = Components.classes["@mozilla.org/appshell/window-mediator;1"]
.getService(Components.interfaces.nsIWindowMediator);
var enumerator = windowMediator.getEnumerator(null);
var count = 0;
while (enumerator.hasMoreElements()) {
var myWindow = enumerator.getNext();
count++;
}
alert(count);