To update a running app which uses a DLL, you have to write an installer that kills or ends the running exe, uninstall the exe including the DLL and then installs the new exe and dll.
You can for example write a cab file setup.dll that will stop/kill the exe and unregister the DLL within install_init. Possibly add code to delete the old registry entry. CE Setup DLL is an option to run code before and after init of install and un-install during a CAB installation.
You may also write code inside an application that does these exe and dll replacement including the exe killing etc. Then include this updater exe in the startup process and run it once.
You cannot rely on starting an app from inside a ce_setup.dll if the app is not already installed.
See http://msdn.microsoft.com/en-us/library/aa924308.aspx for ce_setup.dll description