Continues Delivery, build once or build per environment

Question

We are having a debate in our teams at the moment around the best process for building our software. My team is following gitflow and building once at the start of the deployment pipeline and promoting that build through each environment.

The other team is building once per environment except for prod where they promote the build from Pre-PROD to PRO. They have a branch per environment and when they want to move a feature to a higher level environment they merge the code to the correct branch and trigger a build.

I feel very strongly(!) that my approach is correct, I've read a few books on CI/CD and they almost all start with the concept of build one and promote. However I'm getting a fair bit of resistance to my approach and want to sound out both approaches to a wider audience. The points against raised so far are:

• A defect fix can't be put straight into a higher level environment and has to go through the full pipeline (I think this is a good thing!)
• You can't guarantee that the master branch wont reflect what is in production (I don't fully understand why not as we merge the release branch to master when deployed)

Answer 1

So if I understand correctly, the approach of your team is to create binaries initially, then push to test, preproduction, and eventually production with those same binaries, and the approach of the other team is to use branches to reflect the various environments and perform continuous builds from those branches?

Consider that while I agree with you, as these things always go, there are points for and against this idea. If you push your idea, you should be aware of problems of your approach as well as the points in your favor:

Pros:

• Program behavior does not change, you're guaranteed of that. If what changes are configuration files or database, then you can guarantee a fix for something that works in preproduction simply by ensuring those configuration files and database parameters are the same (no small advantage).
• This approach is slightly more robust. I'm assuming the branch system too will have its share of configurations and database parameters that it uses, however it can also alter the branch code itself to apply fixes. This has its advantages, though contrary to the build once system, it can make finding the source of the problem in production a royal pain in the assessment since anything can be different.

Cons:

• Production bugs are troublesome for quick fixes. Assuming the problem requires an actual change to the code, you must recompile and push through test to preproduction and from preproduction to production. This is not entirely a bad thing mind you, as it is safer, though management tends to completely disregard safety in favor of the quick fix, and you may experience awkward moments with management where you're having to explain that the client will have to wait until you've fully tested the fix before releasing. I do not recommend it.
• Unit testing is very much a possibility as much as it is in the build-per-environment solution, though it is somewhat more awkward to setup. You cannot simply use binaries in other words, you still need to probably tag that version in the trunk of the source repository and use that as a reference.

It seems to me that the critical points here are robustness and safety vs release efficiency and which is more important for your company. You could probably make a good argument that if shortcuts are allowed using the build-per-environment system (and you know good and well they will likely wind up directly updating the production branch for quick fixes), that eventual problems will cost the company more in future problems than it would have cost the company to initially dedicate more time to a fix, and it certainly wouldn't hurt the company's reputation to retroactively prevent bugs.

If this fails, you could humor the possibility of a build directly to preproduction for a quick fix in case of emergencies which would allow testing to still be performed, without simply jump-starting production environment.

Most importantly, it's not worth creating friction for this issue, so be flexible and listen to the other points. Ultimately you only want what's in the best interest of the company, and offering too much resistance is not very productive to the company and won't reflect well on you in any case.

Hope that helps!