How is continuous integration organized in large companies?

Question

In my company, it's common not to do any intermediate build to check how each feature/bugfix branch is merged in dev. There is only daily build, which always elicit a lot of test fails and build errors. I have been told that it's unreasonable to do build for each merge for over 1000 developers.

So I searched how CI is organized in companies that have that much developers or more (Microsoft, Facebook), and found nothing. Maybe, insiders can tell me then?

Answer 1

It is, basically, a scaling problem. You separate your work into modules, which can be different projects and/or different functionalities of your product.

You would have teams that cover sets of those modules. Each of those teams would have CI cycles set up for their scopes, and only after their respective cycles would pass, the code would be pushed to master repos, where the master CI cycle would be run.

The master CI cycle will most likely differ from team level CI cycles in these aspects:

• Team level CI cycles do not have to build the code of entire company, just those modules that they are responsible for and the dependent modules. If there are two modules that are completely independent and in different teams, they would not be part of the other team's CI cycle.
• Team level CI cycles can have much more detailed automated tests than master CI cycle. Master CI cycle would have sanity check tests and regression tests that would, depending on the size of the master solution, be run daily or even weekly, as these tests can sometimes take more than 24 hours to execute.

What you must do with this approach is to provide automated push from local repos to central repo once the local CI cycle passes, lest your developers spend enormous amounts of time to push the code to central repos.

Answer 2

In addition to what @Vladimir_Stokic said, on some teams (mine has ~150 developers), we do builds more frequently than every 24 hours. Whenever a commit occurs, we start a 5 minute timer. After the 5 minutes are up, all commits that happened during the 5 minute interval are combined and built. The build is usually an incremental build. We have a separate builder that runs unit tests for each build that occurs. After the build is finished, if there were any more commits during the build (which takes between 1 and 45 minutes depending on what changed), any pending changes are built, and so-on. We also have a nightly (clean, full) build, but the builds that happen on each commit (roughly) tell us very quickly whether any tests fail.