@vcsjones is correct... the proper way would be with NPAPI or a protocol handler. However, there is another method.
In the past, when I have needed to execute native code with a browser extension, I embed a small HTTP server into a native application that can be called from the browser extension. This allows me to write whatever I need to, while the browser extension is nothing but a control for that native application. You could then easily write something to control OpenVPN.