Why wouldn't my team want to run unit tests automatically?

Question

I'm managing a team in .net. They are writing unit tests, use them locally on a regular basis, and love it. However, they keep pushing to have the unit tests kept as a separate project and only want to run them on a communal build server once a week or so.

In contrast, our JS teams are running tests on the build server on every pull request.

I keep trying to explain to them the benefits of having the tests near the code, and running the tests often, but they keep bringing up fears of the builds being too slow, or what happens when the tests prevent an urgent build.

It's boggling my mind and I'm wondering if there is something I'm missing.

What are likely reasons for this kind of pushback? They are happy to write the tests and they tell me they have close to 100 tests already written on branches they refuse to merge into their main branches until "we come to a decision." ( I.e. I tell them it's ok to keep tests in an isolated project far away from the main code.)

The team was quick to adopt writing unit tests and they find them useful and continue to write more. I am now trying to get them to run the tests in the CI and that is where I'm hitting resistance.

Answer 1

Firstly, thank you to @Gnat for finding insight into the real issue here and giving me what I needed to know to get to the real issue.

We have agreed to isolate the unit tests and integration tests into their own projects to ensure that the added test code doesn't get compiled into the build.

We are also going to be running the tests in a non-blocking workflow until we are able to separate out the code into smaller repositories so that each build doesn't take 15-30 min.

And for now, we have agreed to not build the project on every Pull Request(PR), but instead to run the tests on a nightly build. This is what they really meant by the tests slowing things down... They actually meant doing a build on every PR. They do PRs dozens of times a day. Their main concern about tests blocking an urgent build, was really their concern that urgent builds often end up with dozens of PRs, when everyone is trying to merge their code at once, and they wanted to be able to just run the tests at the end of that process, and not require a separate build at every point along the way.

There wasn't one single answer which gave me all the information I needed, and most of the best answers came in comments, so I've created this answer to summarize up everything that made the most sense.

The important pieces of information I was missing were:

• If the tests are too close to the code under tests, then its very time consuming to tell .net not to build the test files into the .dlls, and it's much easier to just keep them as separate projects.
• The real issue was the build required to make the tests run. It wasn't the tests they didn't want to run, it was the build. They didn't want the build to prevent them from being able to merge code as frequently as they do. Locally, on their own computers, they are able to do a debug build and run the tests in under a minute, but on the CI servers, it takes much longer.

Answer 2

I'm managing a team in .net.

Doesn't matter.

They are writing unit tests ... and only wants to run them once a week or so.

Unit tests are not run once a week. You run them before and after writing the tiniest amount of code you can to make them pass. Tests are usually good but whatever these are they ain't unit tests.

and love it.

Good.

However they keep pushing to have the unit tests kept as a separate project

Doesn't matter.

Our js teams are running tests on every commit.

That's all? Either you commit very often or you don't test much. I test locally first.

What's been left out of this story is how often you're refactoring the tests. Are these tests some entrenched guardian that takes herculean effort to change? Refactoring tests should be made as easy, and as often, as refactoring code.

I keep trying to explain to them the benefits of having the tests near the code, and running the tests often, but they keep bringing up fears of the builds being too slow, or what happens when the tests prevent an urgent build.

They should be allowed to put the tests anywhere they can work. They also should know how to localize a build to the classes that they actually touched so that this is a non issue.

It would be crazy to take tests designed to run "once a week" and shove them into continuous integration. Instead have them add new tests designed to run in the continuous integration that will be tuned to the performance needed there.

The fancy term for these new tests is: continuous integration tests. They are also not unit tests. The best way to categorize tests is based on the performance you need from them. Not the tools you use to run them. That way tests that behave the same way are together.

Do not take control of the tests away from the developers. Give them tools that let them control new kinds of tests.

Answer 3

but they keep bringing up fears of the builds being too slow, or what happens when the tests prevent an urgent build.

What happens if you should prevent an urgent build because you're about to commit some buggy code, but don't because you didn't run the tests that would tell you that you're introducing bad code?

It seems like your team is looking at the tests as an obstacle rather than a tool. The tests are their safety net, their canary in the coal mine. If the builds are too slow, then perhaps that's telling them something they need to know about the software. If they continue to sweep the slow build problem under the rug, it will continue to be a problem.

Answer 4

Firstly, separate projects for unit tests are good. Keeping the unit test code away from the main code is separation of concerns and all that kind of cool stuff.

Secondly, there is no getting away from the fact that unit tests, if run as part of the build will slow things down and queuing for builds can happen. You don't mention what you're using to build (TFS, TeamCity, Jenkins etc) but there are things you can do to mitigate this like multiple queues and priorities. Speak to your build manager.

If there is an urgent build that has to jump the queue, the build manager again should be your first port of call. They may be able to can lower priority builds and push the urgent build higher up the queue.

Finally, you mention CI but don't mention whether it is gated. That is: new code is kicked out in its entirety if the build or unit tests fail. This can undoubtedly be contentious as the code can in theory work, but some conditions may cause the tests to fail.

There is alas, no silver bullet here. When I was a build manager, I often encountered huge resistance when code got kicked out - especially if a deadline was looming. How often I heard the refrain: "TFS keeps crashing. Just turn the setting off and let me check the code in!". Without exception the developer was at fault. They'd either forgotten to include a file in the check in or simply failed to update the test when the code changed as it would have taken too long.

You seriously need to get into the habit of pushing back on the developer here otherwise they will run roughshod over you for ever. If another developer pulls broken code that you allowed through just because another developer starting whining, you will not be popular.

Answer 5

I believe its because of the typical "reluctance to change" problem we study in software engineering all the time. I can't think of a benefit to keep Unit Tests separate (I have worked in JS, Java, .Net and ABAP) but I can see the usual reason, "we have been doing it for a long time and it always worked, so need to change".

Hope that helped. i would also be interested to know if there is a genuine reason.

Answer 6

they keep pushing to have the unit tests kept as a separate project

Tests should go in a separate project. This is standard practice

mySolution
    myProject
    myProject.Tests

and only wants to run them once a week or so.

This is a little odd, although if they are UI tests you might run them overnight. However. Just agree and get the tests checked in.

You can have the 'why don't we run them on checkin?' and 'why don't we not allow merges when tests fail' conversations at a later date.