It is failing because you are sending a message that contains a pointer across a process boundary. Note the fact that you pass an address:
LRESULT insertButtonResult = SendMessage(hWndToolbar, TB_INSERTBUTTON, 0,
(LPARAM)&buttonToAdd);
That final parameter is an address in your processes address space. But the recipient is a different process and the address you pass has no meaning in the address space of the other process.
Some messages, for instance WM_SETTEXT
, will have their payloads marshalled to the other process by the system. But TB_INSERTBUTTON
does not fall into that category. One of the rules of TB_INSERTBUTTON
is that the pointer you pass has meaning in the process that owns the recipient window.
You can resolve this by using VirtualAlloc
, WriteProcessMemory
, etc. to allocate and write to memory in that other process.
Be warned that this is a somewhat difficult task to get right. In particular it matters greatly whether or not the two processes have the same bitness. The layout of the struct differs between 32 and 64 bit. The easiest way for you to make sure you send the right layout is to compile your process with the same bitness as the target process.
By far the easiest way to do something like this is to be inside the target process. If you were to write a plugin then you would not have to deal with any of these issues and would also be able to use officially supported APIs for extension.
As Raymond says, what you are attempting is rather dangerous and you would do well to heed his advice.