What are ways to manage secrets in a big organisation?

Question

Many articles on the internet focus on methods to keep secrets outside public-access (like mobile apps, website frontend etc). But let's take the example of a really big organisation. It has secrets outside public-access, as recommended, but it is within any developer's reach because they have access to the backend code.

What happens if a malicious developer decides to steal the secret (say, an API key) and use it for malicious purposes? Is there a way to store secrets in such a way that a backend developer doesn't have direct access to the API Key?

A solution I thought would work is having a dedicated service that issues the secrets to VM instances, and a CI/CD mechanism for the developers to deploy backend code to production without the backend developers having access to the server. However, I am unsure about the security and drawbacks of this approach.

Answer 1

It is exactly the same as managing secrets in a small company. As soon as you have a company with more than one person in it, you need to manage secrets.

So what can you do to prevent, for instance, the API keys to leak?

1. You audit who access what. If recently a developer requested access to an API key, and a few days later, the API key was used outside the organization, it doesn't prove that the developer is the one who caused the API key to leak, but (1) management may still want to ask a few questions to this developer, and (2) the sole fact that the access is audited may be enough to prevent the secrets to leak in the first place.

2. You restrict access to the API to a range of IP addresses. So given an API key, the API can be used from your company, but not outside. Figuring out who's maliciously using the resources from one of your companies PC could be relatively easy.

3. You don't give access to everybody. If there is a team using an artificial intelligence API, and another team using geopositioning API, you don't give access to both APIs to every member of those two teams. One has access to the first API only; the other has access to the second API only.

   Similarly, there may be specific persons who can access production, and therefore read API keys used in production, and then there can be developers who can't do that. This way, those developers can have their personal API keys, or a shared API key with very limited impact. Say your production API key allows to use some beefy EC2 machines which cost you a few thousands of dollars per month; a shared development API key allows only smaller instances which cost about a few tens of dollars per month.

4. You file a complaint when the API key is stolen. If one of the employees is using it from home, the law enforcement would simply request the IP address to the API provider, and then find the person.

5. Or you make it useless to steal the API key in the first place. If you're pretty sure your employees would be inclined to play with the artificial intelligence API, or store some personal data on Amazon S3, just give them their personal API key that they can use in any way they want. Not only this would make your company look friendlier, but it could also encourage your employees to try something new on their free time, and then bring their inventions back to your company.