debugging a resource leak in a printer driver

Question

I'm trying to debug a memory leak in a printer driver. I'm pretty sure it's a resource leak, not just a plain memory leak because analyzing heap with !heap -s in windbg doesn't show any increase. How do I monitor other kinds of objects with windbg? Number of GDI objects and open handles is not growing either, so what could it be?

The symptom of memory leak is that private bytes are growing up to 180Mb and then printing starts experiencing random problems.

Answer 1

It could be direct calls to VirtualAlloc. Try to isolate the memory type using !address -summary. Better yet, find a copy of vadump.exe from an old resource kit. It gives a more readable breakdown.

You might get some clues by diffing the output from two runs of WinDbg's !vadump command, and then dumping some of the newly allocated RAM. If you have symbol files and you dump RAM using the dps command, WinDbg will display symbol matches for each DWORD. In other words, if a value has a name in the symbol file, you'll see it. A nice example of that is when dumping C++ objects with VTables. The VTable has a symbol, so you'll see what type it is.

Last but not least, you could set a breakpoint on VirtualAlloc and dump the stack for each call. Even absent a rigorous comparison between allocs and frees, you might notice an interesting call stack or allocation size. The breakpoint syntax to dump the stack and continue is

bp kernel32!virtualalloc "kb;g"

In addition, specify a breakpoint on VirtualAllocEx. AFAIK, most process-initiated VAD allocations should hit the break point, except those that are implemented in the kernel, such as file mappings (CreateFileMapping/MapViewOfFile) and maybe LoadLibrary.