Why doesn't my hook DLL work when installed from a service?

Question

I have these functions declared in my DLL. If I use them in a standalone application, all goes well, but when using them in a service, I don't get any result. There's no hook.

function HookProc(code: Integer; wParam: WPARAM; lParam: LPARAM): LRESULT; stdcall;
begin
  Result := CallNextHookEx(Code, code, wParam, lParam)
end;

procedure StartHook; stdcall;
begin
  HookHandle := SetWindowsHookEx(WH_GETMESSAGE, HookProc, hInstance, 0);
end;

procedure StopHook; stdcall;
begin
  UnhookWindowsHookEx(HookHandle);
end;

The issue is only with Vista and up.

Answer 1

You are trying to hook an application on a different desktop. The documentation for the dwthreadId paramteter of SetWindowsHookEx tells you that you cannot do this:

The identifier of the thread with which the hook procedure is to be associated. If this parameter is zero, the hook procedure is associated with all existing threads running in the same desktop as the calling thread.

Remember that session 0 isolation in Vista up means that your services run in a different session (and hence a different desktop) from any interactive desktops.

I imagine that there are other reasons that will stop this working from session 0. You are going to need to run this code from a process that lives in the same desktop as the processes that you wish to hook.