Process hangs in CRT_INIT when exiting

Question

I have a console app that sometimes hangs when it exits. Maybe one out of a thousand times, it will hang. When it hangs, there is one thread left and it consumes 100% of a CPU.

If I look at the process in ProcessExplorer, when I go to the "Threads" tab of ProcessExplorer, my process exits normally.

Looking at the hung process in WinDbg, the call stack looks like:

.  0  Id: 830.d00 Suspend: 1 Teb: 000007ff`fffde000 Unfrozen
  Start: MyHost!COM+_Entry_Point <PERF> (MyHost+0x71ee) (00000000`002571ee) 
  Priority: 0  Priority class: 16384  Affinity: ff
Child-SP          RetAddr           Call Site
00000000`003ddc50 000007fe`ec3f7646 System_EnterpriseServices_Wrapper_7feec3f0000!CRT_INIT+0x65
00000000`003ddc90 000007fe`f91a4485 System_EnterpriseServices_Wrapper_7feec3f0000!DllMainCRTStartup+0x126
00000000`003ddcd0 000007fe`f93f3281 mscoreei!CorDllMain+0x1f5
00000000`003ddd50 000007fe`f93f32cf MSCOREE!ShellShim__CorDllMain+0xe1
00000000`003ddd80 00000000`77a54371 MSCOREE!CorDllMain_Exported+0x37
00000000`003dddb0 00000000`77a54180 ntdll!LdrShutdownProcess+0x1db
00000000`003dded0 000007fe`f91926d2 ntdll!RtlExitUserProcess+0x90
00000000`003ddf00 000007fe`f9192848 mscoreei!RuntimeDesc::ShutdownAllActiveRuntimes+0x294
00000000`003de1f0 000007fe`f89cc919 mscoreei!CLRRuntimeHostInternalImpl::ShutdownAllRuntimesThenExit+0x14
00000000`003de220 000007fe`f897ef51 clr!EEPolicy::ExitProcessViaShim+0x69
00000000`003de270 000007fe`f8976776 clr!SafeExitProcess+0x9d
00000000`003de4e0 000007fe`f8d56971 clr!EEPolicy::GetFinalAction+0x9a
00000000`003de510 000007fe`f7b5f93b clr!SystemNative::Exit+0x52
00000000`003de560 000007fe`99142057 mscorlib_ni+0xcaf93b
00000000`003de630 000007fe`f87f07f3 0x7fe`99142057
00000000`003de900 000007fe`f87f02fe clr!CallDescrWorkerInternal+0x83
00000000`003de940 000007fe`f87f03c7 clr!CallDescrWorkerWithHandler+0x4a
00000000`003de980 000007fe`f892cdf0 clr!MethodDescCallSite::CallTargetWorker+0x2e6
00000000`003deb30 000007fe`f892d54e clr!RunMain+0x1e7
00000000`003ded00 000007fe`f892d448 clr!Assembly::ExecuteMainMethod+0xb6
00000000`003deff0 000007fe`f892cfc2 clr!SystemDomain::ExecuteMainMethod+0x45e
00000000`003df5b0 000007fe`f892cf0e clr!ExecuteEXE+0x3f
00000000`003df620 000007fe`f8929914 clr!CorExeMainInternal+0xae
00000000`003df6b0 000007fe`f91874e5 clr!CorExeMain+0x14
00000000`003df6f0 000007fe`f93e5b21 mscoreei!CorExeMain+0xe0
00000000`003df740 00000000`7782652d MSCOREE!CorExeMain_Exported+0x57
00000000`003df770 00000000`77a5c521 KERNEL32!BaseThreadInitThunk+0xd
00000000`003df7a0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

How can I figure out what this is doing?

Answer 1

This hang was caused by exiting the application before the WorkflowApplication was finished. We were exiting after receiving the "Completed" event but, we needed to wait for Completed and Unloaded.

It was difficult to track down because the hang wasn't in our code and it wasn't in Workflow Foundation code. It was a loop, waiting for a counter to decrement to zero but, nothing was going to decrement the counter. We had Microsoft looking at process dumps, they could see why it was hung but could not find what sequence of events led to the hang because it would only hang 1 out of 1000 times and when we inserted instrumentation, it would not hang.

We finally found the cause by putting a delay just before the return and we realized that we were getting the Unload event that we didn't see without the delay.