You have three options:
- Return a return value known to MSI, assuming you're using InstallShield's Managed Code Custom Action and processing the return codes. (The most common codes there are 0 for success, 1602 for user cancel, and 1603 for failure.)
- Throw an exception, same assumption, which will be treated as ERROR_INSTALL_FAILURE.
- Set a property and use another custom action (such as a type 19 Error action) to exit the installation.
You can combine the first and third approaches to set properties that tweak the messages that will show on the end dialog corresponding to the return code you select. The dialog is chosen per the negative sequences of the InstallUISequence table (SetupCompleteSuccess, SetupCompleteError and such).