Monorepo vs multi repo for large project with multiple partial deliveries

Question

I am looking into options for smoothing out our deliver and release pipeline, and would appreciate some advice on the best way to structure the source code.

This is a pretty large project, which consists of about 30 microservices. Microservices here is a very loose term, but for the sake of the question is it close enough. Currently all of this is stored in one repo.

The internal workflow is structured as follows:

1 month sprints where all code is committed to a dev branch. At the end of the month all tasks that should be moved into testing are merged into the «next major release» branch and all tasks which are critical/bugs are merged into the «active delivery» branch and tested before being selectively rolled out to customers.

Not all customers want the latest and greatest, as they want as few changes as possible to mitigate risk associated with change.

So at the end of the month we have the following:
1 up to date dev branch
1 partly up to date test/next delivery branch
1 production branch

and this is then rolled out to customers like this:
Customer A
- Installed production branched version 11
* overvridden microservice 3 from production branch 11.4
* overvridden microservice 5 from production branch 11.1
* ...

Customer B
- Installed production branched version 11.6
* overvridden microservice 7 from production branch 11.14
* overvridden microservice 15 from production branch 11.7

Customer C
- Installed production branched version 11.3

Would changing to a multi repo setup better cater to our clearly insane delivery pipeline? I am considering one repo for each service, and then one repo for each customer.

Code repo:
-MicroService 1
-MicroService 2
-MicroService 3

Customer repos:
-Link to MicroService 1 - version 1.5
-Link to MicroService 2 - version 2.5
-Link to MicroService 3 - version 1.7
-Customer specific data files (they are not under version control at all now...)

I think this would greatly improve our workflow, and move us back to the sanety of just having one branch. It should give us better control of what is actually released to an customer, gives us the ability to actually reproduce their enviroment and most importantly it should remove the need for merge day once a month.

I would greatly appreachiate some feedback on this.

Edit: The api is pretty stable between services, but the shared libs are a bit of a mixed bag. The product roughly 25 years old, so the core is stable, but as you can image in great need of some clean up as technical dept slowly has been building up.

I basically have two issues which I am trying to solve/improve

1. Merge day is hell, the production branch can be up to a year older then the dev branch, with only a subset of what is in the dev branched merge into it. If a new feature is added in dev, which should not be in the current version, this is not moved to production. All fixes in dev which uses code from this will then cause huge issues if they need to be moved. My idea is that having multiple smaller repos will make this easier.

2. Say there is a bug in a server for getting customer data. The bug is in some shared lib and I fix it, build the CustDataServer and all other servers which I THINK might be influenced by this, then deploy then on the customer/customers system. This means that next time someone does a bug fix, which happens to use my last bug fix, then they basically unintentionally release that into production without meaning to. Most of the time this is fine, but I feel like we have no control over what is actually running in production. ServerA is build with SharedLibA and SharedLibB, while ServerB is build with the same shared libs, but other versions. I am under impression that a multi repo approach would at least give more control in this case.

I am open for all kinds of ideas on how to clean up this mess.

Edit: I appreciate all the though out responsens so far. I agree that selecting repo style is not the issue here.

Your summary is pretty spot on, appart from the layout repos. That was something I was considering to introduce. Currently client spesific files (db and config files mostly) are stored on the client server and are not under any form of source control.

Regarding the release cycle, I agree with you that releasing more often is the best idea. Also, dont do partial releases. All or nothing. The problem is that the release cycle is outside of my control. I can influence many things, but only things which are within the company. Externaly it is hard to make changes. The reason for the partial and infreqent releases is to mittigate the risk for clients. The idea is that the less that is changes, the smaller is the chance of new bugs. The software is very critical, and bugs could litteraly cost millions per day in damages.

What if we tried doing this the other way, I if I describe the system as something to be created with the contraints that exist, what would be the ideal way to structure it?

• Large code base which is logically split up in several units, but are (tightly) coupled
• Yearly release of new features.
• Features are a mix between local changes and system wide new features
• Bugs need to be fixed asap, if critical within hours.
• Customers to not want full releases, they want as little change as possible. In fact, they often dont want the yearly release and drag it out as much as possible
• Critical software, bug are expensive

What would be the best way to structure the internal workflow to limit the number of pain points?

Answer 1

It seems to me that one vs many repos is not the root of your problem.

It's your branching strategy.

• If you can, switch to a recognised branching strategy such as gitflow.

• If you are writing a bug fix for a problem in production. Make a (hotfix) branch from production. Not your develop or next release branch.

• If your 'next release' branch hasn't been merged for a year. Don't even attempt to merge it. Just let it become the new 'production' branch.

• Stop cherry picking merges. Merge a whole branch or not at all. If your various branches have diverged as much as you imply then you probably need to write that bug fix separately for each branch.

In regards to customers deploying different versions of the micro-services. This should not be a problem and it certainly shouldn't be related to your source control.

Save your versioned deployments separately from your source control. Even if you don't compile to binaries, you need to distinguish between the source code and the product.

Don't deploy by checking out code from source control. Make a zip file with the software + config and a deployment script of some kind. Imagine you have to post it on a cd.

I would expect live environments to have multiple versions deployed to enable zero downtime deployments and the like.

As long as each version is tested and you state which other versions it is compatible with this should not be an issue.

Answer 2

As @DocBrown pointed out in the comments:

The original question

Monorepo vs multi repo for large project with multiple partial deliveries

may be not the real question here.

More in the middle, you write

Would changing to a multi repo setup better cater to our clearly insane delivery pipeline?

which leads to the question:

How to deal with our insane delivery pipeline.

First, I try to gather some facts to better understand your current situation:

• You have one source repository which somehow represents the current state of your product

• Your code is logically split up in several units, which are (tightly) coupled

• You have different branches

  • dev
  • next
  • production
• Several "layouts" (repos) for different customers
• Sprints of 1 months length

You have two major pain points:

1) Merge day

2) More control over what goes into production

---

Of course it is hard to give here a one size fits all / glass ball solution, because even though you give a relatively good description of your status quo, it is perhaps more complicated than the 30000ft view, you gave here.

What I could offer here are some thoughts / ideas:

Overall goal: Having one current codebase for all customers

I) Work on the release cycle

I do not know for what reasons there is such a big drift between the production version and the development version.

Is there any chance to release your product more often. The more often you release, the smaller the drift is, the easier the merge should be.

II) Work on the configurability of your application

From what you write it is not obvious what the differences between versions are. Is it somehow possible to make the application more configurable? So that the customer would always get the current version of the product - but perhaps downgraded functionality for his needs.

Of course, I understand, that, the more configuration options you have the more complexity you have to deal with. This doesn't necessary make your life easier - but I see the upside of having a consistent and easier to deal with / unified codebase as a win in your situation.

III) When you have one codebase, which runs on all of our customers servers, you know what is in production and what not; so the second pain point should vanish.

But:

I honestly do not know how feasible these ideas are and how much effort you have to put in or whether the whole is worth the effort.

Since you have good paying customers and the fact, that your product is running (basically) for 25 years - and will so perhaps for another bunch of years - you should take small steps to win the race in the long run.

As a starting point, you could test, how much effort it is to have every customer running one version of one current service with configurable options. And how much the cost of complexity for configuration is.

---

Regarding your original question:

Stay with a monorepo. If things are as tightly coupled as they seem to be, it will be better.

---

Edit:

Perhaps you have to interpret the word "release" a bit more creatively. Release is a fixed point in your repo where on the release-branch is a working set of every components in as many configurations you have.

That means, that some customers get a release (bi-)yearly others more frequent but the customers within a release window get the same running code. Your goal is then: having at every time a current release ready waiting for roll out.

I understand that the customers are interested in stability and favour small changes because they assume the number of defects introduced is also small.

But on the other hand: The customer expects a working product and as long as you can guarantee that - e.g. with a tight test suite - I see no problem in making big updates. N°1 concern is to have as few (new) defects as possible and fix all (up to the point of the release) known bugs.

Aside: I worked for the last 2.5 years under conditions which were not as hard as yours, but similar. 6 customers, 6 different flavours of the underlying application. The underlying application itself consisted of over 7 separated applications which were in their own repo - but in hindsight and if I had been in the project from the beginning, I would vote for a monorepo for all of it, because of the tight coupling. We had at least monthly releases. But things piled up and releases for everything were too hard. I suggested the strategy of having "releases" independent from having them rolled out or activated, so that all projects did effectively always run on "current".

I found that this strategy brought a bit of relief for the developers.