Effectively tracking changes to configuration from dev to prod

Question

This question takes a Spring Boot service as an example, but it could be any technology.

Assuming the following:

• Environments (dev/QA/prod) are owned by different teams. This means dev must not have access to prod configuration.
• Configuration (let's say application.properties) is externalized, ie not part of the binary
• The same binary/package (let's say service.jar) is deployed in each environment and controlled by automated deployment

While changes to the binary artifact (service.jar) get automatically propagated to each environment, changes to configuration still need manual intervention, which inevitably ends up being de-synchronized in each environment.

For example, let's say dev team adds a few key-value pairs to application.properties in their environment. What would be the best way to record these new keys, so that when the deployment occurs in the ops team they know exactly which keys to add, so the risk of starting the new service and seeing it failed because of a missing key is minimized?

I know there will be manual steps involved, but I would like to know how people deals with this and find the most effective way.

Answer 1

This is mainly a communication problem, but you can make errors less likely by some simple technical and organizational measures. First, you should provide a well, high quality documentation of all the entries in your configuration files, as well as some easily accessible example or "default" configuration file. The example file can be automatically deployed to each environment, since it is not intended to be changed by the prod team directly.

Next, with each new release, provide a changelog, where important changes are documented. Changes to the configuration which could prevent the system from working when they are missing are always important, so make sure the information is there.

For example, lets say dev team adds a few key-value pairs to application.properties in their environment. What would be the best way to record these new keys, so that when the deployment occurs in the ops team they know exactly which keys to add, so the risk of starting the new service and seeing it failed because of a missing key is minimized ?

The best way to reduce the risk of failing is to avoid changing your application in a way it requires the new keys, so the application should be backward compatible to older configuration files whenever possible. Often, your application can behave in a sensible way by providing inbuilt default values for the new keys for the case they are missing.

However, if that is not possible, your system should make it as easy as possible for the prod team to find out why the new service fails to start when the key is missing. There should be clear error message, telling exactly which key is missing in which file, and if necessary where to find the information about the missing key, or a hint or example about a meaningful entry for this key.

If the configuration is complex, and the format changes in a way manual editing becomes error prone, you might also consider to provide tools for editing the configs and for migration to a newer version.

For example, I am using the Firefox web browser, and with each new release (which I get automatically), certain things are added to the local configuration one can inspect on the "about:config" page. This is comparable to the config in your "production" environment. Since the whole configuration is kept strictly backward compatible, I never have to add new keys to the config manually just because a there is a new release of the browser. And for the case I want to change something there (maybe a new entry which was not part of the previous version), I either use the Tools/Options menu, or the "about:config" page, and can find the entry plus some kind of documentation. So I recommend trying to implement your system in a comparable way.

Answer 2

At one place I once worked, they had a similar problem. Production configuration was controlled by the build team, only they had access to it in the code repository. The dev, qa, test, etc... configurations could be seen by every one.

In your example, a developer would update the file locally, check it in to source control, and then someone would request the build team to do a new "config only" build which checked out configuration files and deployed them, but did not recompile or redeploy the entire application.

It was up to team members to update config files for other environments at the appropriate time. When it was time to update for production, the build team had to update the file, via an explicit request from the dev team lead, though usually the request simply looked like "copy the app.config file from QA1 to PROD". Sensitive things such as passwords were in a separate config file, and again, only the build team could access the Production password file. The difference was that the developers usually did not request changes to password files because they wouldn't have had Production passwords. The only time they would ask the build team to update it was when adding a new password for a new service. And even then, the developers probably wouldn't know the Production password, they'd only know the key to add to the file (such as "newService2.password").

The technology used to manage a lot of this was Jenkins. There was also an in-house tool that was used to request and schedule builds through Jenkins.