How to prevent configuration drift in Microservices

Question

When I started working everyone was scared of deployments. Software was deployed infrequently and people were afraid services and machines had been manually “patched” since the last release.

I fixed this by automating deployments and configured all services to be deploy when an application is deployed... so everything is deployed frequently, even if nothing changed.

Now we’re thinking of moving to Microservices because we need to support multiple teams working in a single platform, with different release cadence.

How can I ensure services are deployed frequently enough so that people are not scared of deployments?

Answer 1

How can I ensure services are deployed frequently enough so that people are not scared of deployments?

First of all try to understand why are they afraid of deploying new code into a given environment.

Secondly everybody needs to understand the difference between deployment and release.

Released: A business term that defines functionality being available to an end-user.
Deployed: A technical concern that applies in the domain of the team and means the functionality is introduced in Production.
Deployed doesn’t necessarily mean Released. - Source

There are a lots of factors that can influence your approach:

• Your branching strategy
• Your use of feature toggles
• The level of automated testing
• The maturity of your CI/CD pipeline
• The possibility of rolling back
• etc.

Let me go through the above mentioned ones.

Branching Strategy

In case of git flow you have several branches:

• Feature branch: that's where active development is taking place
• Development branch: serves as an integration branch for feature branches
• Master branch: serves as a release history

In other words master branch is considered as a stable code base and could be safely deployed into a staging environment on a daily basis.

In case of feature branch workflow you do not have a development branch. All feature branches are merged directly into the master. Without properly gated / guarded checks (for example: pull requests) master can contain broken code.

Usage of Feature toggle

If you want to deploy a new feature but don't want to release it then that's where feature flags come into play.

Even though the new binaries are in production environment feature flags do not allow new features for all of the end users. With them you can have control over when you want to release a given feature (and to whom).

Automated testing

If you have only just unit tests then your safety net is not really good in terms of regression. You can break integrations (which worked formerly) even though all your tests are green.

If you have higher level tests like integration (where components integration are tested) and end to end tests (where services integration are tested) then your confidence in your new release can be increased.

If these kind of tests are mostly done by the QA team manually then you should consider to invest time and effort to write higher level automated tests as well.

CI/CD maturity

We can differentiate three different levels:

• Continuous Integration:
  • There are unit, integration, regression (and acceptance) tests.
  • The automated deployment is done against a single environment (usually this is staging). There is no deployment promotion.
• Continuous Deployment:
  • There are all kinds of tests (even smoke and sanity tests).
  • The automated deployment is done against several environments (if all tests pass then the deploy is promoted to the next enivornment) and stops at pre-production.
  • A human intervention is needed to deploy into prod.
• Continuous Delivery:
  • Same as the deployment but without human intervention (approval process).

Rollback possibility

Failure can and will happen. They are inevitable. We are human beings and we can make mistakes. So, even though the confidence that was given by tests bugs can find their way to production.

The best thing what you can do to minimize impact is to revert back to the previous version. There are couple of really good ways to do this. The two most well-known are canary release and blue-green deploy.

In case of Canary release you propagate the new code gradually. For example if you have 4 servers then at the first step you deploy only to one server (25%). If you are satisfied then you deploy it to the second server (50%). And as a final step you deploy it to the rest of the servers (100%).

In case of blue-green deployment you are provisioning new nodes (green) which are identical to the production nodes (blue). You deploy the new code to all of the green nodes. You register these green nodes into the load balancer. You allow traffic to both blue and green. If you are satisfied then you remove the blue nodes from the load balancer. If don't then you remove the green ones.