Security: Preventing Unauthorized Loading/Calls into C++ DLL

Question

I have a C++ DLL, foo.dll, that provides API methods that give cleartext answers to encrypted data. I want to come up with a robust solution to only allow friendlies (eg. user.dll) to load this DLL and use its functions.

This is not a problem of DLL spoofing, but rather a question of how to hobble a DLL to only work in certain contexts. I'm in a potentially non-networked context on multiple platforms, so phoning home or relying on a system call is not a likely solution.

Something I've considered is a home brewed initialization step sharing a 'magic key' for initialization. With security, 'roll your own' is often bad.

Any advice of accepted practices for this kind of DLL level security?

Answer 1

The only way to protect encrypted data is by ensuring that it requires a very long, unpredictable and secret key to decrypt it. The security comes not from obscuring the method of decryption, but the key itself. You cannot prevent someone from calling into your DLL but the functions in the DLL can require that the caller provide a decryption key as a parameter. This way, only clients that know the secret key will be able to decrypt the contents. You can try to cheat by embedding the decryption key somewhere in your DLL, but it can then be found simply by loading the DLL into a debugger. So don't do that.