How do you apply development practices like version control, testing and continuous integration/deployment to system administration?

Question

Imagine you're going to manage a number of servers with a number of different services that's used by a number of people. Now say you want to reconfigure or replace some software on one of those servers. Obviously you don't want to work on servers that are in production.

If this was a code change, as a developer, I would make the change on my local development machine, test it locally and commit the change to a version control system. The changes could then be deployed in a staging environment, tested further and finally deployed in a production environment. It would also be easy for me to roll back, if necessary.

Generally, or specifically, how do you achieve this in system administration?

(The first thing that comes to mind is to use virtual machines and put virtual machine images in version control, but I'm sure there is a lot of literature and clever solutions I'm not presently aware of.)

Answer 1

Use chef or puppet to enforce machine configurations, and place their cookbooks and recipes under version control. Yes vms would make things easier but even physical server provioning can be controlled by kickstart or preeseed which can again be version controlled.