C++ - using 'new' operator cannot be stepped into during debug

Question

COM+ application, building with MS Visual Studio 6, SP 6 on Windows XP SP 3, and debugging remotely.

My main question is this; why would I not be able to step into 'new' if I can step into 'delete'? I'm mostly looking for ideas as to what I should look into.

I'm working on a fairly large project that I'm only just getting familiar with. The current issue is a heap corruption problem in which a release build will eventually exhaust its working set and crash. The problem is so pervasive that the following code will corrupt the heap:

int * iArray = new int [100];
delete [] iArray;

I say 'corrupt the heap' because the debug output displays "heap[dllhost.exe]: invalid address specified to rtlvalidateheap" on the 'delete'.

I can step into the 'delete' call fine and it seems to be calling the proper one (located in DELOP.cpp in ...\Microsoft Visual Studio\VC98\CRT\SRC) but, for whatever reason, I cannot step into any call to 'new'. I'm grasping at straws here but I have a feeling somewhere in the code base someone overrode the 'new' operator and the code I'm looking at is unintentionally using it. The symptoms of the release build seem like memory is being allocated to one heap and attempted to be deleted from another, at least that's my hunch.

EDIT: Ack! Sorry everyone, I posted too soon, should have given more info.

I've searched the code base and found a few overrides but they are all in classes, not globally defined. The only one that isn't in a class is the following:

struct _new_selector 
{

};

inline void* operator new(size_t, void *ptr, _new_selector)
{
    return (ptr); 
}

But that's a placement new and I'm pretty sure it doesn't count in this situation. What is the library that I should be stepping into for the original 'new'? I'm guessing it's the same as the 'delete' but if not, maybe I just don't have debug info for it?

EDIT 2: Messing around with this I've found that on a debug build this issue does not exist. I've already looked into the run-time libraries, it's using /MD and /MDd for debug. Not only that, but I've built the release version with /MDd jut to make sure and there was still no change. Looking at the maps of both a debug build and a release build the new operator (with it's mangling) is at the following:

Release:

0001:00061306 ??2@YAPAXI@Z 10062306 f MSVCRTD:MSVCRTD.dll

0002:00000298 _imp??2@YAPAXI@Z 1006e298 MSVCRTD:MSVCRTD.dll

Debug:

0001:00077d06 ??2@YAPAXI@Z 10078d06 f MSVCRTD:MSVCRTD.dll

0004:00000ad4 _imp??2@YAPAXI@Z 100b5ad4 MSVCRTD:MSVCRTD.dll

I also checked the delete operator:

Release:

0001:000611f0 ??3@YAXPAX@Z 100621f0 f msvcprtd:delop_s.obj

Debug:

0001:00077bf0 ??3@YAXPAX@Z 10078bf0 f msvcprtd:delop_s.obj

Also, and I don't have a print out of them but I can get it if it would help, the disassembly of the new operator looks the same in release and debug. So I guess it's not an override? Would an inline override of an operator make this untrue?

Also, being a COM+ application which spawns multiple dllhost.exe processes, would it be possible for a call to the new operator go to another DLL, or the exe, and a call to delete go to the opposite?

Answer 1

Ok, so here is what it ended up being.

The program I'm working on has about thirty or so projects within it. Some create libs, others dlls, still others exes. In any case, it's all intermingled. Add into that the fact that we use ATL and COM and it starts getting crazy pretty quick. The end result is that some, not all, of the projects were being built with the _ATL_MIN_CRT compiler definition, even though this is a desktop application, not a web based one in which a client would need to download a few modules.

Here is some info on _ATL_MIN_CRT:

• http://support.microsoft.com/default.aspx?scid=kb;EN-US;q166480

• http://msdn.microsoft.com/en-us/library/y3s1z4aw%28v=vs.80%29.aspx

Notice, from the first link, that this also will eliminate the use of memory allocation routines. I’m unsure what the original motive behind using this was, or if it was really intentional, but it certainly causes issues with allocating memory. Also, this only affects Release builds, hence why it was such a pain to find.

Basically, module A was built with _ATL_MIN_CRT and module B was built without it but had a dependency to module A. Since this is also using COM and everything was run in dllhost.exe, when module B tried to use the new operator it seems to have gone out of its dll to attempt to allocate memory on the heap. Then, when calling delete, it tried to delete it within its dll. Thus, we have a crazy memory leak.

Mind, removing _ATL_MIN_CRT fixes this but what I mention above is only my understanding of it. It may very well be more/less complicated but, regardless, this was the issue.

Thanks to everyone's suggestions, they really did help me find this thing!

Answer 2

Going with your hunch that there is possibly an overloaded new somewhere in the code, few things you can check

1. the disassembly of the code and find out the library name in that file, generally there is something in the assembly that will give you a hint
2. If you are not able to find out the library name, check the address in the assembly that you are entering. Then in the debug output window, check the load addresses for the various libraries - that could give you a clue as to which library to check
3. If the above doesn't help, check if you can generate a map file for the complete project. If you can, then you can look up the address in the map file and that might help
4. Try to use the debug version of the runtime libraries. Can't recollect what was the option that will turn on the debug_malloc. That can help you figure out what is happening on the heap

The community can add a few more that I might have missed. And finally, if you do crack the problem, please share how you did so. Either here or as a link to your blog. Working on heap problems for a large project is generally not easy and we all can learn a new trick or two.