Where to store public and private gpg keys?

Question

All security is a trade-off.

(I'm not a crypto/security expert. These are my understandings from what I've read and studied, but if you're doing anything important, get professional advice from someone who's done this a lot for real).

In this case you have a number of choices that differ mainly on how they trade off uptime/convenience with key theft/abuse risk. I'm assuming you're using a GnuPG/OpenPGP library, rather than the command-line tools, but if not "the app" can be considered the GnuPG agent.

Store the key un-encrypted on disk. The app can use the key whenever it wants. If the app is restarted, it has immediate access to the key. An attacker that breaks into the system or steals an (unencrypted) backup can use the key easily. Proper backup encryption is vital.
A marginal improvement over this approach is to store the key encrypted and store the (obfuscated) passphrase for the key elsewhere in the system / in the app binary; it makes life a bit harder for the attacker and means they at least have to spend more time on it, but in most cases they'll still be able to recover it pretty easily. Proper backup encryption is vital.
Store the key encrypted on disk and store it decrypted in memory on app startup. A human can decrypt the key when prompted during app startup; after that, the app can use the key whenever it wants. Theft of the key from disk / backups does the attacker little good, they have to go to the extra effort of recovering the key from the application's memory, or modifying/wrapping the application to capture the passphrase when entered by the administrator after a crash/restart. Key must be locked in memory that cannot be swapped out.
Store the key encrypted on disk and decrypt it only with specific administrator interaction. The app cannot use the key without an admin intervening. The key pretty safe on disk and the theft risk from app memory is limited by the short periods it's in memory. However, an attacker that has broken into the system can still modify the app to record the key when it'd decrypted, or capture the passphrase. Key must be locked in memory that cannot be swapped out.
Store the key on removable storage. Physically insert it on app startup to decrypt the key and store it in app memory like (3), or when the app actually needs to use the key like (4). This makes it a bit harder for the attacker to steal the encrypted key and makes password theft less useful, but no harder to modify the app to steal the decrypted key. They can also just wait until they see the storage inserted and copy the encrypted key if they've stolen the passphrase with a wrapper/keylogger/etc. IMO it's not much benefit over a strong passphrase for the encrypted key on disk - it might make life a little harder on the attacker, but it's a lot harder on the admin.
Store the key on a smartcard, crypto accelerator, or USB crypto device that's designed never to permit the key to be exposed, only to perform crypto operations using it. The PKCS#11 standard is widely supported and useful for this. The key (theoretically) cannot be stolen without physically stealing the hardware - there are key extraction attacks on lots of hardware, but most require lots of time, and often require physical access. The server can use the key at will (if the accelerator has no timeout/unlock) or only with admin intervention (if the accelerator is locked after each use and must be unlocked by the admin). The attacker can still decrypt data using the accelerator by masquerading as the app, but they've got to do a lot more work, and will need to have ongoing access to the target system. Of course, this one costs more.

Disaster recovery is more challenging for this option; you depend on physical hardware for decrypting your data. If the data center burns down, you're done for. So you need duplicates and/or a very securely stored copy of the key its self. Every duplicate adds risk, of course, especially the one plugged into that "just in case" backup server we don't really use and don't keep the security patches up to date on...

If you go for hardware with a key built-in rather than one where you can store but not read the key, you have the added challenge that one day that hardware will be obsolete. Ever tried to get business critical software that requires an ISA card running on a modern server? It's fun - and one day, PCI/X and USB will be like that too. Of course, by then the crypto system you're using might be broken anyway, so you'll need to decrypt all your data and migrate it to another setup anyway. Still, I'd be using hardware where I can generate a key, program it into the hardware, and store the original key in a couple of different forms in a bank safe deposit box.

Now that you've read that, remember: I'm just an interested not-even-hobbyist. Go ask a professional. When they tell you how totally wrong I am, come here and explain :-)

Whatever you do, DO NOT invent your own crypto system.