How to determine the optimal release frequency for maximum throughput?

Question

I work on a large software programme - 100 developers in financial services.

The common wisdom of Continuous Integration is to get feedback early from your changes.

The common wisdom from Continuous Delivery is that by getting good at releasing small chunks, you reduce the risk of failure, because you can roll back easily - and so releasing small chunks helps you deliver to production rapidly and often.

A business value diagram in Lean allows us to see the flow of business value from left to right (similar to a production line) and from this you can identify where your change items are getting stuck, and where the bottlenecks in the process are.

The challenge in software development is identifying precisely what the widgets on the production line are.

If you read The Phoenix Project, then the changes are the change records flowing through the system(although this is heavily IT infrastructure focused). If you talk to a Scrum master - then the changes flow through the system are stories. If you talk to a developer, then the changes that flow through the system are GIT commits. (Which can and should align to stories).

The simple reality is that we do small releases once a month, and large releases once every three months, due to the transaction cost of the regression test. (Don Reinersen's book The Principles of Product Development Flow is amazing on the tradeoffs of cycle time and transaction cost.

So in trying to identify the constraints on the system - instead of finding a work area where the items are piling up - to me it seems that the batch size itself is a constraint. By batch size, I mean the number of deliverables in a release. A release every month with a large number of developers would have a large number of stories/commits. I'm trying to quantify this.

We know that the economics of batch size is a u-curve optimisation problem, and that the transaction cost of a regression test and release is substantial.

My question is: How can I determine the optimal release frequency for maximum throughput?

Answer 1

I've been thinking about this also and to do this, I think you first need a hypothesis. Here's where I would start:

Regression Overhead

You are bound by the minimum time to regression test. I'm going to assert that you can't release faster than that. I think a basic formula for this would be based on the following components:

• Time required to test all unchanged features
• Time required to update/add features
• Time required to test new/changed features

Time to regression grow over time as you add features but the marginal rate of increase will diminish over time. Your situation may vary but I think we can assume this is fixed. If it's growing significantly, having releases on any fixed frequency will not be sustainable.

Defects and Time to Fix

Hypothesis 1

Let's assume that the number of defects found in a release is proportional to the number of features in the release. We'll also assume that the time to fix and retest those defects is proportional to the number of defects.

Hypothesis 2

Again we assume that the number of defects found in a release is proportional to the number of features in the release. But now we assume that the time to fix and retest these is some function of the number of defects that is not a straight multiple. the reasoning is that if you find some defect and fix it, then regression testing must be run again. These defects prevent some other defects from being discovered. This creates a lead time on fixing defects.

Testing the Hypotheses

In order to determine whether either of these match reality, I think you need to determine/guess the bare minimum time to regression test everything. You can't go faster than that. Then you need to try different release schedules. This might require some spikes. Then you start trying to decompose regression time for the number of features. You might see something like this:

Or maybe it looks like this:

Where the green is your regression baseline time. Or maybe it looks like something completely different. Based on my experience it's more like the latter.

If it's the former, it would suggest that you should go with really long release cycles because that minimizes overhead of the baseline regression. However there's another factor that you haven't mentioned: business value. If a feature produces value for the company (and why are you doing it if it doesn't) the longer it takes to put it into production the less value is adds. So even if it is a straight-line cost, you need to consider that.

Once you understand this relationship, then I think you will most likely want to make the dev time for a set of features match the QA time for those feautres (considering other overhead activities like fixing defects, updating tests, sharpening the saw, etc.)