Why does calling ProcessGroupPolicyEx callback cause an access violation?

Question

I'm trying to help a colleague with some code in a client side extension. Since adding in a call to the callback, the function seems to complete ok, but an event in the Windows Event log complains about an access violation whilst processing the group policy object.

After removing existing code, with just the added call to the callback, it still reports this access violation.

Can you please help identify what we might be missing?

//
// Entry point for processing group policy objects.
//
// For full details, see http://msdn.microsoft.com/en-    us/library/windows/desktop/aa374383(v=vs.85).aspx.
//
extern "C" DWORD CALLBACK ProcessGroupPolicyEx (
  __in   DWORD dwFlags,
  __in   HANDLE hToken,
  __in   HKEY hKeyRoot,
  __in   PGROUP_POLICY_OBJECT pDeletedGPOList,
  __in   PGROUP_POLICY_OBJECT pChangedGPOList,
  __in   ASYNCCOMPLETIONHANDLE pHandle,
  __in   BOOL *pbAbort,
  __in   PFNSTATUSMESSAGECALLBACK pStatusCallback,
  __in   IWbemServices *pWbemServices,
  __out  HRESULT *pRsopStatus)
{

 if(pStatusCallback)
   pStatusCallback (FALSE, L"Aaaaargh!");

   return (0);
}

This code has been tried using a static string, an array of bytes on the stack, a array of bytes that's been new'd and deliberately leaked - in case the method was taking ownership of the memory. Also been CoTaskMemAlloc'd, just in case. All produce the same problem.

The (redacted) error in the eventlog is:

Windows cannot process Group Policy Client Side Extension Exception 0xc0000005.

To make things interesting, this is just on some OS's, fully patched XP 32bit is one of the definite problems. 2008R2 works fine.

Yes - we need it to work on XP 32bit.

Other weird behaviour that may have a bearing here: If we call this function multiple times, it fails on the 3rd call. No exception is thrown, no text is shown, none of our code after the call is executed, no additional errors in the event log. Timing is not a factor here: it happens if you call it 3 times in a row, or 3 times over 5 minutes. This does not happen if we wrap the calls in a generic try/catch block. No exception is caught - all the text is shown. All the code is run. We still get the error in the event log, however.

Answer 1

Looks like we've found the issue with this.

The problem is that the callback needs to be made with __stdcall calling convention. By default, visual studio creates projects with the __cdecl calling convention. If you add the /Gz flag to your project, it will use __stdcall by default. We couldn't do that, however, since we're pulling in other modules with different calling conventions.

The underlying problem is that UserEnv.h defines the callback like this:

typedef DWORD (*PFNSTATUSMESSAGECALLBACK)(__in BOOL bVerbose, __in LPWSTR lpMessage);

This is a strange definition. All other windows callbacks are defined like this:

typedef INT_PTR (CALLBACK* DLGPROC)(HWND, UINT, WPARAM, LPARAM);

That CALLBACK is important, it expands like this:

#define CALLBACK    __stdcall

This means that by default, all windows callbacks are defined to use __stdcall calling conventions, except this one, for some reason.

If we create our own callback defintion:

typedef DWORD (CALLBACK *PFNSTATUSMESSAGECALLBACK_STDCALL)(__in BOOL bVerbose, __in LPWSTR lpMessage);

And assign our function pointer to it:

PFNSTATUSMESSAGECALLBACK_STDCALL pStatusCallback = (PFNSTATUSMESSAGECALLBACK_STDCALL)pRawStatusCallback;

Then we can use the pStatusCallback function pointer with the __stdcall calling convention and have things work properly.