Track Data Execution Prevention (DEP)

Question

When running one of our software, a tester was faced with the data execution prevention dialog of Windows.

We try to reproduce this situation on a developer computer for debugging purposes : with no success.

Does anyone know how to find what may cause the DEP protection to kill the application? Is there any existing tools available for this?

Answer 1

The DEP dialog will typically only show when you try to execute code from a region that you're not marking as executable. This might be caused by 'thunks' in a library you're using, e.g. ATL windowing. This problem is fixed in ATL 8.0.

A stack-trashing bug - for example, a buffer overrun - can also cause this problem, by setting the return address to a location that isn't executable. This might not cause an access violation but instead weird behaviour, if DEP is turned off for the process or not available on the hardware.

It might also happen if you throw a C++ exception or raise an SEH exception, and your structured exception handlers have been trashed by a buffer overrun.

Answer 2

Potentially I would think any time you try to write to memory that isn't allocated this would be a possible result. Could be anything along the lines of deleting an object then using it, or writing a string into a buffer which is too small to hold it.

Answer 3

DEP is influenced by the presence of hardware capability. We recently had a situation where our app ran fine on old machines, but would fail on new ones. It turned out that although DEP was enabled on both the old servers and the new servers, we crashed on the new ones because the hardware detection was better, more aggressive, or something like that. So if your QA can reproduce but the DEV can't, then try it with identical hardware... Although it seems unreasonable that the QA would have a newer/better PC than the dev... I totally believe it!
Here are my notes on our recent experience with this: Incompatibilities between Indy 9 and Windows Server 2003?