Instead of using a BHO, the proper extension for this sort of task is a MIME Handler. See http://msdn.microsoft.com/en-us/library/aa767916(v=vs.85).aspx for details on this topic.
Having said that, all IE extensions (and URLMon pluggable extensions in particular) are EXTREMELY difficult to write properly and as a consequence are a frequent source of security holes and reliability bugs. They also don't work in other browsers, and don't work in the "Metro" mode of IE10+ on Windows 8+.
Rather than going the browser extension route, you would be FAR better off just registering your MIME type/file extension in the Windows Registry and having that invoke your application. This is stable and works across all browsers and in all modes.