SetWindowsHookEx returns 0 when compiling for the .NET 4.0 framework in 32bit machines

Question

I'm trying to set a low level windows keyboard hook to grab three keys pressed even if the application is not in focus. To do this I'm calling SetWindowsHookEx as

// Create an instance of HookProc.
KeyboardHookProcedure = new HookProc(KeyboardHookProc);
//install hook
hKeyboardHook = SetWindowsHookEx(
    WH_KEYBOARD_LL,
    KeyboardHookProcedure,
    Marshal.GetHINSTANCE(
        Assembly.GetExecutingAssembly().GetModules()[0]),
    0);
//If SetWindowsHookEx fails.
if (hKeyboardHook == 0)
{
    //Returns the error code returned by the last unmanaged function called using platform invoke that has the DllImportAttribute.SetLastError flag set. 
    int errorCode = Marshal.GetLastWin32Error();
    //do cleanup
    Stop(false, true, false);
    //Initializes and throws a new instance of the Win32Exception class with the specified error. 
    throw new Win32Exception(errorCode);
}

This used to work on 32bit and 64bit machines using the .NET Framework 3.5 but after upgrading to the .NET Framework 4.0 stopped working on 32bit machines.

Does anyone know how to solve this so that I can use the 4.0 Framework and make this work on both 32bit and 64bit machines?

Answer 1

import the dll like this:

[DllImport("kernel32.dll")]
    public static extern IntPtr GetModuleHandle(string name); 

then use

GetModuleHandle(Process.GetCurrentProcess().MainModule.ModuleName)

to replace

Marshal.GetHINSTANCE(
    Assembly.GetExecutingAssembly().GetModules()[0]

Answer 2

From the documentation for SetWindowsHookEx

hMod [in]
HINSTANCE
A handle to the DLL containing the hook procedure pointed to by the lpfn parameter. The hMod parameter must be set to NULL if the dwThreadId parameter specifies a thread created by the current process and if the hook procedure is within the code associated with the current process.

So you should be passing in IntPtr.Zero for NULL

//install hook
  hKeyboardHook = SetWindowsHookEx(
    WH_KEYBOARD_LL,
    KeyboardHookProcedure,
    IntPtr.Zero,
    0);

Answer 3

Solved it by targeting each platform separately. Configured VS to compile both a Win32 and a Win64 version and deploying on x86 and x64 machines their corresponding binary.

The Win32 or x86 runs on both 32bit and 64bit machines.

Answer 4

Hans Passant:

Any module handle will do since it doesn't actually get used for low-level hooks, no DLL needs to be injected to make them work. Some care in selecting one is required for .NET 4 since its CLR no longer fakes module handles for pure managed assemblies. A good one to use is the one you get out of pinvoking LoadLibrary("user32.dll") since it is always already loaded. You don't have to call FreeLibrary().

You'll need this declaration to call LoadLibrary:

[DllImport("kernel32", SetLastError=true, CharSet = CharSet.Auto)]
private static extern IntPtr LoadLibrary(string fileName);