The options i see are:
- tell the user to restart the browser after installation/update
- tell the user to close the browser(s) before proceeding with installation
Note that you still may have issues replacing the DLL. What e.g. Flash does is suffixing their filename with the version (e.g. npYourPlugin_1_2_77.dll), so file locks are no issue.
If you do that you might not even need to restart the browser when you can call navigator.plugins.refresh()
(works since Firefox 17, not sure about other browsers though).
Also note that i'm not sure about the ActiveX plugin (IE) if you need to support that as well - maybe taxilian will chime in on that and other scenarios i haven't run into myself.