Cannot modify params in EnumWindows callback

Question

I try to get the main window handle using following code in Lazarus (Free Pascal):

function FindMainWindow(Pid: LongWord): LongWord;
type
  TParam = record
    Window: HWnd;
    Test: Integer;
    Pid: LongWord;
  end;
  PParam = ^TParam;
var
  Params: TParam;
  function _FindMainWindow(Wnd: HWnd; MyLParam: PParam): Bool; stdcall;
  var
    WinPid: DWord;
  begin
    with MyLParam^ do
    begin
      Test := 2;
      GetWindowThreadProcessID(Wnd, @WinPid);
      Result := (WinPid <> Pid) or (not IsWindowVisible(Wnd))
        or (not IsWindowEnabled(Wnd));
      if not Result then begin
        Window := Wnd;
      end;
    end;
  end;
begin
  Params.Pid := Pid;
  Params.Test := 1;
  EnumWindows(@_FindMainWindow, LParam(@Params));
  ShowMessage('Done!');
  ShowMessage(IntToStr(Params.Test));
  Result := Params.Window;
end; 

The Problem is that Params.Test is still 1 after running the callback. I want to modify the Params in the _FindMainWindow function.

Note: I could not access Params in _FindMainWindow directly because i get an "Access violation" error.

Answer 1

Certainly in Delphi, and it appears FPC too, nested functions are not valid for use as callback functions. When using the 32 bit compiler, it so happens that nested functions can be used as callbacks. But such code is only accepted by the compiler because the callback functions are declared in the Windows unit as untyped pointers. If the Windows unit declared them as procedural types, you find that the compiler objects to using nested functions.

For the 64 bit Delphi compiler, you simply cannot use nested functions as callbacks at all. Whilst the compiler lets you go ahead, because of the use of untyped pointers in the Windows unit (see above), the callback functions are not called correctly. Apparently that is true for FPC too. You will have to stop using nested functions for your callbacks.

It's interesting that both FPC and Delphi compilers have the same characteristics here. My guess is that the x64 calling convention, which is a register based convention as opposed to the stack based x86 stdcall is the driving force behind this issue. I bet that if you tried to use a nested x86 register function as a callback, then that would fail at runtime too.