Bizarre hook behavior 32/64 bit

Question

I'm using a local hook (WH_KEYBOARD) with ms word (OpusApp). Well, as far as I know a 32bit app with a 32bit DLL must work only with 32bit target applications. The weird thing is that the program only works with 64bits apps!!! That is it, only with 64bits APPS! For example, it works with IE 64 but not with IE 32! The app and dll are 32bit compiled with radstudio XE2, I confirmed the version into PE header. In 32bit OSs, the app and dll doesn´t work.

I found no solutions on net and see no starting point to solve this weird problem.

The DLL code:

// Exported functions

extern "C" __declspec(dllexport)bool __stdcall InstallMouseHook(unsigned long, void *);

extern "C" __declspec(dllexport)bool __stdcall RemoveMouseHook();

// Callback Procedure Declaration

LRESULT CALLBACK HookProc(int code, WPARAM wParam, LPARAM lParam);

// Global variables

HHOOK HookHandle;
HINSTANCE DllInstance;
typedef void (__stdcall *CALLIT)(int,WPARAM,LPARAM);
CALLIT callIt = NULL;

int WINAPI DllEntryPoint(HINSTANCE hinst, unsigned long reason, void*)
{
  DllInstance=hinst;
  return 1;
}

bool __stdcall InstallMouseHook(unsigned long pid, void *function)
{

     callIt = ( CALLIT ) function;

     if (function == NULL) {

        ShowMessage("function is null!");

     } else if (callIt == NULL) {

        ShowMessage("callIt is null!");

     }  

     HookHandle=SetWindowsHookEx(WH_KEYBOARD ,reinterpret_cast<HOOKPROC> (HookProc),DllInstance,pid);

    if (HookHandle==NULL)return false;

    else return true;

}

bool __stdcall  RemoveMouseHook()
{
  if(UnhookWindowsHookEx(HookHandle)==0)
  {
    return false;
  }
  else return true;
}

LRESULT CALLBACK HookProc(int code, WPARAM wParam, LPARAM lParam)
{
    if (code<0) {
        return CallNextHookEx(HookHandle,code,wParam,lParam);
    }

    if (callIt != NULL) {
        callIt(code,wParam,lParam);
    } else {
        ShowMessage("HookProc - no function to execute OR 32/64 bits problem!");
    }

  //Call the next hook in the chain
  return CallNextHookEx(HookHandle,code,wParam,lParam);
}

The EXE calling code:

void __fastcall TfrmMouseHook::btnHookAppDllClick(TObject *Sender)
{
    HWND hWindow;
    unsigned long pid;

    String s = "MouseHookDLL.dll";
    DllHandle=LoadLibrary(s.w_str());
    MOUSEHOOKFCT_2 InstHook=reinterpret_cast<MOUSEHOOKFCT_2> (GetProcAddress(DllHandle,"InstallMouseHook"));

    hWindow = FindWindow(ComboBox1->Text.w_str(),NULL);

    if (!hWindow) {
        msg("hWindow fail");
        return;
    }

    pid = GetWindowThreadProcessId(hWindow ,0);
    if (!pid) {
       msg("pid fail");
       return;
    }

    if(!InstHook(pid, (void *) callIt )) {
        msg("Unable to install  hook!");
    } else {
        msg(" #### hook INSTALLED! ####");
    }


}

CALLIT callIt(code,wParam,lParam) {
    frmMouseHook->msg("hook callit: code="+IntToStr(code) +" wparam="+IntToStr(wParam)+" lparam="+IntToStr(lParam) );
}



   Call IT is a function pointer to a hooker app function.

    Any ideas will be very wellcome!

Answer 1

What happens? Besides MSDN or anyone else says, there are some bug in XE6, compiling DLL in newer versions of IDE make this behavior disappears, in fact, the new DLL crash and hook nothing.

As Remy noted, by test I passed a function pointer to the DLL, one wrong thing to do but, when added with the wrong thing done by Embarcadero, sort of functioned.

By now, and I know people will get mad, I put both methods (the wrong and the correct hooks) in the same DLL and in my application and... get crazy... could hook into 32 and 64 bits app with only one DLL.

Don't believe? Install XE6 and try!

And Works in windows 10 as well in windows 7.

Answer 2

It is physically impossible for a 32-bit app to install a 32-bit hook DLL and have it executed in 64-bit processes. A 32-bit DLL simply cannot be injected into a 64-bit process. Period. MSDN says this in multiple places, including in the SetWindowsHookEx() documentation:

SetWindowsHookEx can be used to inject a DLL into another process. A 32-bit DLL cannot be injected into a 64-bit process, and a 64-bit DLL cannot be injected into a 32-bit process. If an application requires the use of hooks in other processes, it is required that a 32-bit application call SetWindowsHookEx to inject a 32-bit DLL into 32-bit processes, and a 64-bit application call SetWindowsHookEx to inject a 64-bit DLL into 64-bit processes. The 32-bit and 64-bit DLLs must have different names.

Because hooks run in the context of an application, they must match the "bitness" of the application. If a 32-bit application installs a global hook on 64-bit Windows, the 32-bit hook is injected into each 32-bit process (the usual security boundaries apply). In a 64-bit process, the threads are still marked as "hooked." However, because a 32-bit application must run the hook code, the system executes the hook in the hooking app's context; specifically, on the thread that called SetWindowsHookEx. This means that the hooking application must continue to pump messages or it might block the normal functioning of the 64-bit processes.

If a 64-bit application installs a global hook on 64-bit Windows, the 64-bit hook is injected into each 64-bit process, while all 32-bit processes use a callback to the hooking application.

The fact that you say your app and DLL do not work on 32-bit OS versions suggests your hooking code is flawed to begin with. But you have not shown enough code to diagnose that one way or the other.