Definition of brittle unit tests

Question

I've seen the terms brittle and fragile used to describe poorly written unit tests. But I haven't been able to find a clear definition for what brittle or fragile actually mean in this context.

• Is it unit tests that break for more than one reason?
• Is it when a test fails because something in the code base is changed that is supposedly unrelated to the test?
• Is it tests that test functionality in an inappropriate manner, such as testing how instead of what a function does?
• Something else entirely?

Or are these terms simply a cover-all for all unit-test smells?

Answer 1

The terms "brittle" and "fragile" do not have a clear and specific definition within the context of software development. Rather, they are ordinary English words that are sometimes used in software development to describe certain kinds of code.

In the context of software development, these words mean exactly the same thing as they do everywhere else: "brittle" and "fragile" things are things that are easy to break.

That said, there is this "personal reflection essay" from Wikipedia. If you're looking for an "official" definition, this one is as good as any:

Software brittleness is the increased difficulty in fixing older software that may appear reliable, but fails badly when presented with unusual data or altered in a seemingly minor way.

Things that make unit tests brittle:

1. Asserting against elements in a UI.
2. Asserting against large result strings instead of discrete values.
3. Static state or improper state sharing between threads.
4. Unrealistic or otherwise inadequate test data.
5. Elaborate mocking schemes.
6. Brittle code under test.
7. Too many edge cases or unreasonably high cyclomatic complexity.
8. Too many responsibilities.
9. Leaky abstractions.
10. Code under test is not idempotent between test sessions.
11. Tests whose results can vary based on the environment, such as performance timings.
12. Tests that can't run independently of each other.

As you can see, what makes unit tests fragile is often not the fault of the test, but the fault of the code being tested.

One way to have better tests (and by extension, better code), is to practice Test-First Development. Writing your tests first forces you to think about the surface area of your code's API, and strongly encourages you to write that surface area in a way that makes it simpler, more robust, more deterministic and easier to test.

Further Reading
The Brittleness of Unit Tests
Some Good Advice for Newcomers to Unit Testing