You mean that you want to run a custom action that references a function within a dll that is installed with the package? In this case use the custom action type 17. Or in WiX:
<CustomAction Id="myCAfromInstalledDLL" FileKey="IdOfFile.dll" ExeCommand="EntryPointInDll" />