Here's a question on ServerFault that provides some good details on what specifically you can do with PEM (there's quite a bit more than I think would be prudent to copy/paste).
As far as self-signing, yes this is something you can do, although not trivially. In additional to the work setting it up there will be some ongoing maintenance that can be a real pain, especially if you don't know what you're doing very well. The problem is really two-fold:
- Your clients have to install the root CA cert of your VPS or you have to install it for them. This is somewhat invasive and will require an administrator. Additionally, if your root CA ever changes (which it should at least as often as it expires) you have to update all the machines again.
- You assume substantial liability for the security of the system. If your VPS is compromised somehow, whether it is exploited/penetrated/spoofed is all the same, an attacker can impersonate your executable to the client. As you can imagine, this can have catastrophic consequences.
Self-signing is generally not advised for production environments, especially with outside customers. There is too much to know and too many ways to screw it up.
If cost is an issue, you should check out Comodo's code signing certificate offerings. They are generally the best priced and quite reliable. They were hacked a copule years ago, but there was a lot the security field learned from that incident, and IMO it wasn't really Comodo's fault.