Why are released ArcObjects COM objects not finalized until the GC is manually forced to do so?

Question

I am working on a .NET application for processing geographic data that uses ESRI's ArcObjects COM library via ESRI's own .NET interop assemblies.

When running in production, the process can crash during some operations due to reaching the 2GB per-process memory limit. (ArcObjects is a 32-bit library.) This is because some of the processing steps can create many temporary ArcObjects geometry objects. It will leak memory and eventually run out of memory despite manually releasing these objects using FinalReleaseComObject and associated helper methods. However, I can force the GC to release the memory by calling WaitForPendingFinalizers, and calling it regularly along with GC.Collect and FinalReleaseComObject keeps the memory usage under control. Otherwise, many objects remain in memory until the process exits (normally or abnormally).

First question: why isn't the memory held by the ArcObjects COM objects released immediately? Or, why does the GC allow the process to crash instead of finalizing the released COM objects and reclaiming the memory before it crashes?

The application runs in production on Windows 2008 64-bit, while I develop using Windows 7 32-bit. I can get the process to crash on the production boxes, but not on my development box. I thought that this may have been because locally I usually run in Visual Studio with a Debug build, but I have also tried it without the debugger (Start Without Debugging) using a Release build but even then it didn't use anywhere near as much memory as in production and won't crash.

Second question: Why?

EDIT: In my previous experiments, I worked out that GC.Collect by itself is not sufficient, even though I was calling it explicitly. I have a utility method that calls GC.Collect followed by GC.WaitForPendingFinalizers and calling it after every algorithm iteration keeps memory usage down.

Answer 1

COM objects used in managed applications are sitting behind a runtime-callable wrapper (RCW) which is a managed object that replicates the interface of the COM object towards the managed client (it's a bridge between the managed and unmanaged components). If I remember correctly, the actual COM interface references are held by the RCW, not by your code. When you release the COM object, it's really the RCW that's released but that, being a managed object itself, doesn't go away until the GC gets around to clean it up. When that happens, the RCW is deleted and the last reference to the COM object is gone, so it can destroy itself. (According to the documentation, FinalReleaseComObject should set the refernce count to 0 but I've seen similar behavior in the past so I question whether the documentation is correct.)

As for your second question, I have a guess: I've seen in stressed environments that the GC simply doesn't get a chance to run when the system is under heavy load. Back then we determined that the GC ran on a lower-priority thread and our application was using so much CPU that the GC never had a chance to clean up. We had to add another thread that was periodically calling GC.Collect() every once in a while - this forced the GC thread to awaken and do its magic. You may be facing something similar. (This may be the reason behind the your issue in #1, too.)