What I've done before was put all my custom action's logic in an executable and then call the executable, which worked nicely. It turns out that WiX provides various Custom Action Projects within Visual Studio that remove the need to create a stand alone executable, see here for a nice example by James Schaffer.
As for controlling when the custom action is executed, you control this by adding an entry in the InstallExecuteSequence node of the *.wxs file, like this...
<InstallExecuteSequence>
<Custom Action="CheckLogLevel" Before="InstallServices" />
</InstallExecuteSequence>
... where CheckLogLevel
is my custom action, and it's being directed to be run before the InstallServices
action.