How do you deal with configuration files in source control?

StackOverflow https://stackoverflow.com/questions/6009

  •  08-06-2019
  •  | 
  •  

Question

Let's say you have a typical web app and with a file configuration.whatever. Every developer working on the project will have one version for their dev boxes, there will be a dev, prod and stage versions. How do you deal with this in source control? Not check in this file at all, check it with different names or do something fancy altogether?

Was it helpful?

Solution

What I've done in the past is to have a default config file which is checked in to source control. Then, each developer has their own override config file which is excluded from source control. The app first loads the default, and then if the override file is present, loads that and uses any settings from the override in preference to the default file.

In general, the smaller the override file the better, but it can always contain more settings for a developer with a very non-standard environment.

OTHER TIPS

Don't version that file. Version a template or something.

Configuration is code, and you should version it. We base our configuration files on usernames; in both UNIX/Mac and Windows you can access the user's login name, and as long as these are unique to the project, you are fine. You can even override this in the environment, but you should version control everything.

This also allows you to examine others' configurations, which can help diagnose build and platform issues.

My team keeps separate versions of the config files for each environment (web.config.dev, web.config.test, web.config.prod). Our deployment scripts copy out the correct version, renaming it to web.config. This way, we have full version control on the config files for each environment, can easily perform a diff, etc.

Currently I have the "template" config file with an added extension for example:

web.config.rename

However, I can see an issue with this method if critical changes have changed.

The solution we use is to have only the single configuration file (web.config/app.config), but we add a special section to the file that contains settings for all environments.

There is a LOCAL, DEV, QA, PRODUCTION sections each containing the configuration keys relevant to that environment in our config file(s).

What make this all work is an assembly named xxx.Environment which is referenced in all of our applications (winforms and webforms) which tells the application which environment it is operating on.

The xxx.Environment assembly reads a single line of information from the machine.config of the given machine which tells it that it is on DEV, QA, etc. This entry is present on all of our workstations and servers.

Hope this helps.

+1 on the template approach.

But since this question has tag Git, the distributed alternative springs to mind, in which customizations are kept on a private testing branch:

A---B---C---D---           <- mainline (public)
     \       \
      B'------D'---        <- testing (private)

In this scheme, the mainline contains a generic, "template" config file requiring the minimal amount of adjustments to become functional.

Now, developers/testers can tweak the config file to their heart's content, and only commit these changes locally on one a private testing branch (e.g. B' = B + customizations). Each time mainline advances, they effortlessly merge it into testing, which results in merge commits such as D' (= D + merged version of B's customizations).

This scheme really shines when the "template" config file is updated: the changes from both sides get merged, and are extremely likely to result into conflicts (or test failures) if they are incompatible!

I've used the template before, i.e. web.dev.config, web.prod.config, etc, but now prefer the 'override file' technique. The web.config file contains the majority of the settings, but an external file contains environment-specific values such as db connections. Good explanation on Paul Wilson's blog.

I think this reduces the amount to duplication between the config files which can cause pain when adding new values / attributes.

I have always kept all versions of the config files in source control, in the same folder as the web.config file.

For example

web.config
web.qa.config
web.staging.config
web.production.config

I prefer this naming convention (as opposed to web.config.production or production.web.config) because

  • It keeps the files together when you sort by file name
  • It keeps the files together when you sort by file extension
  • If the file accidentally gets pushed to production, you won't be able to see the contents over http because IIS will prevent *.config files from being served

The default config file should be configured such that you can run the application locally on your own machine.

Most importantly, these files should be almost 100% identical in every aspect, even formatting. You shouldn't use tabs in one version and spaces in another for indenting. You should be able to run a diff tool against the files to see exactly what is different between them. I prefer to use WinMerge for diffing the files.

When your build process creates the binaries, there should be a task that overwrites the web.config with the config file appropriate for that environment. If the files are zipped up, then the non relevant files should be deleted from that build.

@Grant is right.

I'm on a team with close to 100 other developers, and our config files are not checked into source control. We have versions of the files in the repository that are pulled with each check out but they don't change.

It's worked out pretty well for us.

I version control it, but never push it to the other servers. If the production server requires a change, I make that change directly to the config file.

It may not be pretty, but it works just fine.

The checked-in, plain-vanilla version of app/web.config should be generic enough to work on all developer machines, and be kept up to date with any new setting changes, etc. If you require a specific set of settings for dev/test/production settings, check in separate files with those settings, as GateKiller stated, with some sort of naming convention, though I usually go with "web.prod.config", as not to change the file extension.

We use a template config file that is checked in to version control and then a step in our automated build to replace specific entries in the template file with environment-specific settings. The environment-specific settings are stored in a separate XML file that is also under version control.

We're using MSBuild in our automated build, so we use the XmlUpdate task from MSBuild Community Tasks to update the values.

For a long time, I have done exactly what bcwood has done. I keep copies of web.dev.config, web.test.config, web.prod.config, etc. under source control, and then my build/deploy system renames them automatically as it deploys to various environments. You get a certain amount of redundancy between the files (especially with all of the asp.net stuff in there), but generally it works really well. You also have to make sure that everyone on the team remembers to update all the files when they make a change.

By the way, I like to keep ".config" at the end as the extension so that file associations do not get broken.

As far as local developer versions of the config file, I always try my best to encourage people to use the same local settings as much as possible so that there is no need to have your own version. It doesn't always work for everyone, in which case people usually just replace it locally as needed and go from there. It's not too painful or anything.

On our project we have configuration stored in files with a prefix then our build system pulls in the appropriate configuration based on the current system's hostname. This works well for us on a relatively small team, allowing us to apply config changes to other people's files if/when we add a new configuration item. Obviously this definitely doesn't scale to open source projects with an unbounded number of developers.

We have two problems here.

  • Firstly we have to control the configuration file that is shipped with the software.

    It is all two easy for a developer to check in an unwanted to change to the master config file, if they are using the same file in the devolvement environment.

    On the other side, if you have a separate configuration file that is included by the installer, it is very easy to forget to add a new setting to it, or to let the comments in it get out of sync with the comments in the devolvement configuring file.

  • Then we have the problem that developers have to keep there copy of the configuration file up-to-date as other developers add new configuration settings. However some settings like database connection strings are different for each developer.

  • There is a 3rd problem the question/answers do not cover. How do you merge in the changes a customer have make to your configuration file when you install a new version of your software?

I have yet to see a good solutions that works well in all cases, however I have seen some partial solutions (that can be combined in different combinations as needed) that reduces the problem a lot.

  • Firstly reduce the number of configuration items you have in your main configuration file.

    If you don’t have a need to let your customers change your mappings, use Fluent NHibernate (or otherwise) to move the configuration into code.

    Likewise for depency injection setup.

  • Split up the configuration file when possible, e.g. use a separate file to configure what Log4Net logs.

  • Don’t repeat items between lots of configuration files, e.g. if you have 4 web applications that are all installed on the same machine, have a overall configuration file that the web.config file in each application points to.

    (Use a relative path by default, so it is rare to have to change the web.config file)

  • Process the development configuration file to get the shipping configuration file.

    The could be done by have default values in the Xml comments that are then set in the configuration file when a build is done. Or having sections that are deleted as part of the process of creating the installer.

  • Instead of just having one database connection strings, have one per developers.

    E.g first look for “database_ianr” (where ianr is my username or machine name) in the configuration file at run time, if it is not found, then look for “database”

    Have a 2nd level "e.g. -oracle or -sqlserver" make it quicker for developers to get to both database systems.

    This can of course also be done for any other configuration value.

    Then all values that end in “_userName” can be striped out before shipping the configuration file.

However in the end what is you is a “owner of configuration file” that takes the responsibly of managing the configuration file(s) as above or otherwise. He/She should also do a diff on the customer facings configuration file before each shipment.

You can’t remove the need for a caring person some this short of problem.

I don't think there's a single solution that works for all cases as it may depend on the sensitivity of data in the config files, or the programming language you're using, and so many other factors. But I think it's important to keep the config files for all environments under source control, so you can always know when it was changed and by whom, and more importantly, be able to recover it if things go wrong. And they will.

So here's how I do it. This is for nodejs projects usually but I think it works for other frameworks and languages as well.

What I do is create a configs directory at the root of the project, and under that directory keep multiple files for all environments (and some times separate files for each developer's environment as well) which are all tracked in source control. And there is the actual file that the code uses named config at the root of the project. This is the only file that is not tracked. So it looks like this

root
|
|- config (not tracked)
|
|- configs/ (all tracked)
    |- development
    |- staging
    |- live
    |- James

When someone checks out the project he copies the config file he wants to use in the untracked config file and he is free to edit it as he wishes but is also responsible to copy these changes before he commits to the other environment files as needed.

And on servers, the untracked file can simply be a copy (or reference) of the tracked file corresponding to that environment. In JS you can simply have 1 line to require that file.

This flow may be a little complicated at first but it has great advantages: 1. You never have to worry about a config file getting deleted or modified on the server without having a backup 2. The same if a developer has some custom config on his machine and his machine stops working for any reason 3. Before any deployment you can diff the config files for development and staging for example and see if there's anything missing or broken.

We just keep the production config file checked in. It's the developer's responsibility to change the file when they pull it out of source safe for staging or development. This has burnt us in the past so I wouldn't suggest it.

I faced that same problem and I found a solution for it. I first added all the files to the central repository (also the developer ones).

So if a developer fetches the files from the repository the developer config is also there. When changing made to this file, Git should not be aware of these changes. That way changes cannot be pushed/committed to the repository but stay locally.

I solved this by using the git command: update-index --assume-unchanged. I made a bat file that is executed in the prebuild of the projects that are containing a file whose changes should be ignore by Git. Here is the code I put in the bat file:

IF NOT EXIST %2%\.git GOTO NOGIT
set fileName=%1
set fileName=%fileName:\=/%
for /f "useback tokens=*" %%a in ('%fileName%') do set fileName=%%~a
set "gitUpdate=git update-index --assume-unchanged"
set parameter= "%gitUpdate% %fileName%"
echo %parameter% as parameter for git
"C:\Program Files (x86)\Git\bin\sh.exe" --login -i -c %parameter%
echo Make FIleBehaveLikeUnchangedForGit Done.
GOTO END
:NOGIT
echo no git here.
echo %2%
:END

In my prebuild I would made a call to the bat file, for example:

call "$(ProjectDir)\..\..\MakeFileBehaveLikeUnchangedForGit.bat" "$(ProjectDir)Web.config.developer" "$(SolutionDir)"

I found on SO a bat file that copies the correct config file to the web.config/app.config. I also call this bat file in the prebuild. The code for this bat file is:

@echo off
echo Comparing two files: %1 with %2
if not exist %1 goto File1NotFound
if not exist %2 goto File2NotFound
fc %1 %2 
if %ERRORLEVEL%==0 GOTO NoCopy
echo Files are not the same.  Copying %1 over %2
copy %1 %2 /y & goto END
:NoCopy
echo Files are the same.  Did nothing
goto END
:File1NotFound
echo %1 not found.
goto END
:File2NotFound
copy %1 %2 /y
goto END
:END
echo Done.

In my prebuild I would made a call to the bat file, for example:

call "$(ProjectDir)\..\..\copyifnewer.bat" "$(ProjectDir)web.config.$(ConfigurationName)" "$(ProjectDir)web.config
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top