How to prevent CI/CD process from being suspended while big changes happen?

Question

This is more of a procedure and maybe a philosophical problem.

The development team I am in is divided into Scrum teams or Squads that work on the same product. Occasionally, either one of these squads would have to embark on something 'big' such as a revamp of a core functional or component, database upgrade, etc.

The current approach adopted by the team is to put a lockdown on the current release, and have only the work for that squad merged for the nightly build. Other squads still pursue their sprint objectives but are prohibited from merging the tickets into the nightly build, therefore preventing qualification of the tickets. Essentially, every other work is blocked except for the 'big' change.

What are our options to handle or at least manage the situation but in a more efficient way? It might also be worth noting that squads are geographically apart.

Answer 1

This is exactly the problem Hewlett Packard had when they re-architected their solution. You can read about it in this book (short version).

Basically, you should architect your transition to allow both old and new version to coexist during the transformation. The problem with this is that it makes things more complicated and take more effort. And if often requires architecture in place to allow for such gradual transition. But it is is possible more often than developers and architects are willing to admit.

Really, taking big architectural change and transforming it into chain of small, gradual changes is core of agile, and I would say engineering, skillset. You should just remind them that you are fine with possibly increased cost of softer and more gradual transition.

It is difficult to give you specific answer, given different changes require different approaches and solutions to make them more gradual. Having both old and new version at the same time is often the case. Backwards compatibility is also often seen in these scenarios. Feature flags are also a good idea.

Related :

• https://martinfowler.com/bliki/BranchByAbstraction.html
• https://martinfowler.com/articles/evodb.html