How to override an application's single instance limit in Windows?

Question

I am trying to override the singe instance limit of an application for which I don't have the source. I know that the app is using the good ol' trick of using CreateMutex to determine whether there is another instance running. (If the mutex is created successfully it proceeds, if getlasterror says that the mutex has been created it quits immediately). I found that through sniffing the Win32 api calls. I thought using Detours would do the trick, but it doesn't quite work out. I am intercepting CreateMutexW, but for some reason, it doesn't catch the first four calls to it. (Again I know what these calls are by sniffing win32 calls and looking at the name of the mutexes). I do get the fifth one intercepted, but the one I actually want to intercept is the first one.

I am using detours through the sample application withdll. I wonder if the problem is that detours is kicking in too late or because of some kind of protection these calls may have. Is detours the best approach? Perhaps using something else may be a better idea?

Answer 1

There might be several reasons for the situation you describe. Here are the most probable of them:

1. The CreateMutexW call you need to catch occurs within the DllMain method of one of the DLLs that are imported by the process, and you are using the DetoursCreateProcessWithDll() function to inject your code. Detours injects your DLL by placing it at the end of the process executable import list, and hence all the DLLs that are imported by the process would be loaded and initialized within the process prior to yours. In order to overcome this, try using CreateProcess(CREATE_SUSPENDED) and CreateRemoteThread()-based injection, although this method raises its own challenges.
2. The API that is used in the first call is different. Have you tried overriding CreateMutexExW? Are you sure ANSI methods call Unicode ones?

Hope this helps.