That's not the correct way to use CreateMutex()
. CreateMutex()
(almost) always succeeds in returning a valid handle to a mutex. From the MSDN doc which you linked: "If the mutex is a named mutex and the object existed before this function call, the return value is a handle to the existing object, GetLastError returns ERROR_ALREADY_EXISTS..."
Quoting again: "Two or more processes can call CreateMutex to create the same named mutex. The first process actually creates the mutex, and subsequent processes with sufficient access rights simply open a handle to the existing mutex. This enables multiple processes to get handles of the same mutex, while relieving the user of the responsibility of ensuring that the creating process is started first."
What you want to do is use CreateMutex()
to open a handle to the mutex, and then use WaitForSingleObject()
with a timeout value of 0 to try to take it. If the WaitForSingleObject()
fails to take the mutex, then you return MY_ERROR_BUSY. If it does succeed to take the mutex, call ReleaseMutex()
when you are done using it to unlock it.
ETA:
If WaitForSingleObject returns WAIT_OBJECT_0 or WAIT_ABANDONED, then you own the mutex (e.g. it becomes signalled) and you must call ReleaseMutex to give-up ownership (e.g. unsignal it) before calling CloseHandle.
If it returns WAIT_TIMEOUT, you do not own the mutex, and you can just call CloseHandle.
int FunctionExposedByDll()
{
HANDLE hMutexAPI = CreateMutex(0, 0, API_RUNNING_MUTEXT );
int rval = MY_ERROR_BUSY;
if ( hMutexAPI )
{
DWORD wait_success = WaitForSingleObject( hMutexAPI, 0 );
if ( wait_success == WAIT_OBJECT_0 || wait_success == WAIT_ABANDONED )
{
// actual function here
ReleaseMutex( hMutexAPI );
rval = SUCCESS_VALUE;
}
CloseHandle( hMutexAPI );
}
return rval;
}
You should take some time to get more familiar with the basic interprocess communication primitives, starting here: http://en.wikipedia.org/wiki/Mutex
ETA once more I figured C++ 11 may have added the IPC primitives to the standard library, and it appears they have:
http://en.cppreference.com/w/cpp/thread/mutex,
http://en.cppreference.com/w/cpp/thread/lock_guard
If you're using a C++11 capable compiler (e.g. Visual Studio 2013), please use these instead of the system primitives.
And one more edit... As pointed out in a comment, the C++ standard library primitives are not usable for interprocess communication, only for synchronizing threads in the same process. (This is a great disappointment.)
Instead, use Boost.Interprocess if you can.