It looks like you can run a .exe from Firefox using nsIProccess
https://developer.mozilla.org/en-US/docs/Code_snippets/Running_applications so you should look into that.
however, it is not advisable to run a .exe from browser, as this can make your site look malicious.
I couldn't find anyway for chrome. however, you can load external libraries using the NPAPI, this may work for exe as well but I have no clue: http://developer.chrome.com/extensions/npapi.html