when should unit tests be done in a continuous integration environment

Question

We are trying to adopt some CI practices and methodologies into our organization. I'm currently reading over some of the points in "Continuous Delivery" book written by Jez Humble and David Farley. And the point that I am debating is when unit tests should be run - before or after git commits. The book seems to suggest running unit tests after you check in, as a part of a commit test suite. But wouldn't it make more sense to conduct unit tests before checking in?

Any comments would be appreciated.

Answer 1

Usually the developer runs them manually before checking in, and a CI server runs them automatically after checking in. Programmers are usually pretty good about running unit tests for incremental builds on the configuration they've been working on, but they don't do things like do a clean build from a completely clean checkout from version control, run tests for all configurations of a product, run all the unit tests for the downstream projects that depend on theirs, etc.

That's why CI servers "rerun" the tests after commit. You can configure them to run the tests before they actually get merged into a master branch, if you want a branch that definitely has had all the tests run.

Answer 2

Running your automated tests on your CI Server is actually a form of Integration testing.

When a developer runs tests locally they are validating whether the changes they have made mesh with the state of the entire system when they checked out their code.

Your CI Server, on the other hand, is continually integrating work from all developers. When it runs automated tests it's to validate that developer changes mesh together correctly.

Answer 3

Committing broken tests / broken code to your local repo is not problematic in and of itself. Problems start to arise when one locally commits to a specially-designated integration branch in anticipation of a push to a central repo. Real problems do arise when one pushes broken stuff to a specially-designated branch on a central repository.

A system that completely precludes committing broken tests is a broken system. If you follow the mantra of writing tests before writing code, the tests will necessarily fail at first. Making it impossible to commit those intentionally broken tests does not make sense.

A system that completely precludes committing broken code is also a broken system. Developers may start with pseudo code that doesn't even compile, then proceed to real code that doesn't work, and finally to real code that apparently works. Making it impossible to commit those initial ideas does not make sense. Having my broken code committed to my local repo has saved my rear end more than once.

Stealing words from the Zen of Python, "Configuration management is one honking great idea -- let's do more of those!"

I tend to work in esoteric environments where complete local testing is impossible (e.g., supercomputers, networks of powerful non-supercomputers, embedded devices, and networks of embedded devices, none of which I have on my desk). This makes me a bit overly leery of overly aggressive hooks in a CM system.

Even if you don't work in such esoteric environments, having a commit hook that automatically triggers testing for every single commit (and then rejects the commit in the case of a failure) is just a bad idea. It's best to encourage people to commit frequently and to merge frequently. Merging in a branch that contains uncommitted code is a recipe for a good amount of configuration management angst. Failing to merge frequently is a recipe for even more configuration management angst.

The opposite extreme, having no hooks at all and then making developers do embarrassing stuff when they break the build, is also a bad idea. What's needed is a compromise that empowers people to be their most efficient and yet protects against people doing the stupid things that even very smart people are wont to do.