Some basic assumptions you have to make here about how a mutex works on Windows:
- a mutex is an operating system object that's reference-counted. It will not disappear until the last handle on the mutex is closed
- any handle that's left unclosed when a process terminates is closed by the operating system, decrementing the reference count
- a mutex is re-entrant, calling WaitForSingleObject on a mutex on the same thread succeeds and needs to be balanced with an equal number of ReleaseMutex calls
- an owned mutex becomes abandoned when the thread that owns it terminates without calling ReleaseMutex. Calling WaitForSingleObject on a mutex in this state generates the WAIT_ABANDONED error return code
- it is never a bug in the operating system.
So you can draw conclusions from this by what you observed. Nothing happens to the mutex when A crashes, B still has an handle on it. The only possible way B can notice that A crashed is when A crashed while it owned the mutex. Very low odds for that and easily observed since B will deadlock. Far more likely is that B will happily motor on since it is now completely unobstructed, nobody else is going to acquire the mutex anymore.
Furthermore, a deadlock when A starts back proves something you already knew: B owns the mutex permanently for some reason. Possibly because it acquired the mutex recursively. You know this because you noticed you had to call ReleaseMutex twice. This is a bug you need to fix.
You'll need to protect yourself against a crashing sibling process and you need to write explicit code for that. Call OpenProcess on the sibling to obtain a handle on the process object. A WaitForSingleObject call on the handle will complete when the process terminates.