Release strategy for multiple Git feature branches being tested simultaneously
https://softwareengineering.stackexchange.com/questions/365809
-
29-01-2021 - |
Question
We have three environments: Dev
, UAT
and Prod
. We use TFS
to schedule releases of our master
branch into Dev
for our internal verification, then UAT
for business verification, and of course finally to Prod
once approved.
We've recently adopted a new lightweight Git branching strategy as follows:
master
is always prod-ready. At any point master
should be able to be deployed to production.
All new development is done in a separate feature (topic) branch as follows:
- Create a new feature branch off
master
, call itFeatureA
- Develop
FeatureA
until completion - Once
FeatureA
is finished, release it toDev
, and thenUAT
- Once the business signs off on
FeatureA
inUAT
, it's considered prod-ready. MergeFeatureA
intomaster
, then deploy the newmaster
branch toDev
thenUAT
. During the way, "smoke test" the branch inUAT
to ensure the resulting merge into master didn't cause any unforeseen side-effects. Once smoke-tested, release toProd
.
The problem we're coming across right now is that we may have multiple features being developed in parallel, all of which could potentially need to be deployed to the test environment for verification at the same time. The approach we've taken to solving this problem is:
If FeatureA
and FeatureB
need to be in UAT at the same time, then:
- Create a new branch,
FeatureAandB
, which will encompass both features - Merge
FeatureA
intoFeatureAandB
- Merge
FeatureB
intoFeatureAandB
- Release
FeatureAandB
toDev
, thenUAT
The downside to this is that it's unlikely both FeatureA
and FeatureB
will be UAT
verified at the same time. If FeatureA
is verified and FeatureB
is not, we need to release FeatureA
to prod without FeatureB
. What we've discussed in this scenario is to:
- Merge
FeatureA
(not the joint branch, but just FeatureA) intomaster
- Release
master
toDev
, thenUAT
for a quick smoke-test, and finallyProd
- Once in prod, re-release just
FeatureB
toDev
thenUAT
so testing can continue.
The downside to this is that it directly impacts any testing for FeatureB
, and potentially unwinds any work the testers have accomplished with FeatureB
.
How do you manage multiple features living simultaneously in each environment and being released potentially independent of one another? We can mitigate the issue a little more if we have multiple environments, or turn-around UAT testing much quicker, but at the end of the day the same problem can exist.
I'm not opposed to hearing alternative branching strategies, either.
Solution
Ahh. Congratulations! You’ve destroyed one bottleneck and discovered the next one! Now it’s time to look at actually continuously integrating your code. As you’ve found out, it’s hard to continuously deliver when your code under dev isn’t being continuously integrated, but how do you make this work?
No more feature branches.
No. Seriously. No more feature branches.
It’s time to introduce feature toggles into your system. Instead of delivering new features by delivering code, you need to have complete control over when a feature is turned on for a given environment. In other words, you want to have environment specific configuration that turns features on and off. This decouples code releases from feature releases.
OTHER TIPS
Test features separately and in order, that's really the best way to do it. Usually whichever one is done first gets tested first, but business needs could change that priority. When FeatureA
is accepted, it's merged into the main branch (dev
, UAT
, or master
in your case, as applicable), that branch is merged into FeatureB
and then feature B is tested. A is tested first, then B, then C, etc, with each feature branch getting updated as other features are tested and accepted.
By testing features separately, you keep the focus on one feature and one team is responsible for fixing any problems. There's a clear responsibility for keeping the code in a good state when handling merge conflicts (in this case, the featureB
team needs to merge dev
in and deal with any merge conflicts, ensuring that the code from featureA
is maintained). Separate features also makes it easy to assign "blame" when a problem is discovered - not the "why did you break this" kind of blame, but the "this feature is causing a problem, remove it from dev
until we can sort it out" kind of blame
If you're finding that a lot of FeatureX
branches are spent waiting for testing or multiple branches are being completed at the same time, you can either break them up into smaller features, or allocate more resources for testing. Both approaches should decrease the time to test an individual branch.
Similar features that will be completed at the same time may best be developed on a single branch. That's kind of what you propose with the FeatureAandB
branch, but in this case, they get accepted or rejected as one. If the features are not similar / related, though, test them separately.
Finally, make sure you're making use of any automation available, so teams aren't waiting on another team for anything. Most build servers can be set to build Pull Request branches, producing a FeatureA+dev
build which can go to QA for acceptance and integration testing. The developer simply opens the PR (which also serves as a code review, if desired, and also serves to signal that the feature is ready for testing), and once QA signs off on it, completes it. This then merges the branch into dev
automatically. Can do the same for UAT
and master
, creating PRs for merging dev
into UAT
, and UAT
into master
.
You may then be able to reduce the number of branches, and stick with dev
and master
, plus featureX
. Get rid of UAT
, as that role is essentially served by dev
. Actual development is done on featureX
and production is in master
. Deploy new code whenever master
is updated, or deploy whatever is in master
whenever you want to release.