It seems under Windows 64 bit, the application must be also 64 bit one (not sure for 32 bit OS).
My system is Windows 7 64 bit and if I build my app with "Target Platform" set to "64-bit Windows", the following works.
::ShellExecuteW( this->Handle, NULL, L"openfiles.exe", L"/query", NULL, SW_SHOWNORMAL );
I believe other launching options also would work like CreateProcess.
But gives the same error you when built as 32-bit application.