Does BDD pay off?

Question

I a bit familiar with rspec [Ruby] and specs [Scala]. Yesturday I passed a tutor for Cucumber. What I disliked about Cucumber is that additionally to describing scenarios (like you would do with spec- or xUnit-style testing) you have to implement extra layer of indirection: translating scenario steps into ruby expressions. For me creating unnecessary (?) extra layers of indirection feels like "heavy-weight" J2EE-way, not like "light-weight" ruby way. Is understandability by "domain experts" the only advantage of Cucumber? Or is there some non-obvious (technical?) advantages for developer/tester as well?

Answer 1

BDD, from a practical standpoint, is highly synonymous with TDD. Rspec is a BDD test framework, as well as Cucumber.

That stakeholders can read and understand Cucumber acceptance specifications is certainly a key advantage, but this fact alone really doesn't get at the real benefit of Cucumber. Your features and scenarios ought to grow somewhat in specificity as the work being done for them moves through the value stream of your team's development cycle.

Some teams may have an analyst scoping out work at the beginning of the cycle. Sometimes this analyst writes gherkin acceptance specifications, but whoever writes the first draft, you would expect them to be fairly coarse grained. They may not cover every un-happy path.

As the developers take up the work, they often discover edge cases and missing scenarios. At this point they can touch base with the analyst, and the results of such conversations should be written into the cucumber features.

In my experience the testers have cultivated an even more critical eye, and thus it is not uncommon to see them add even more scenarios and features. The testers may also uncover defects, which should be added to the cukes to protect us from regression.

The point is that, in addition to providing executable documentation for our code, Cucumber also provides a repository for the state of the team's conversations vis a vis the features being developed.

There is certainly extra overhead to all of this. However, it's worth considering how much overhead is already in your team's process, which Cucumber might serve to streamline. I find that Cucumber helps reduce the amount of thrash that happens in communication about features both within and outside of the team room.

I should also mention that cucumber is intended as full-stack acceptance testing, and therefore should be less fine-grained, relative to your unit tests. And cucumber is not a good substitute for unit and integration tests. I also would never recommend using cucumber to verify aesthetic aspects of your app's UI. Just use it to validate the actions which a user might take when using your app.

Answer 2

Cucumber is designed to help engage business stakeholders in refining developers' and testers' understanding of the system by collaborating to create scenarios that everyone can understand.

The act of engaging business stakeholders pays off because everyone gets a better understanding, they start sharing the same language and carrying that language into the code (see Domain Driven Design's "Ubiquitous Language"), which can lead to better estimates, appreciation of scope, conversations around options for achieving the same goal, etc. etc.

There are certainly other ways of achieving the goals. For instance, on our C# project we talk through the scenarios, write them on a wiki then implement them using a little custom Domain Specific Language rather like this one. The same thing could be done in Ruby.

BDD is the process of learning the places where we thought we knew what we were doing, but it turned out we were wrong - the discovery of ignorance. With Feature Injection and unit-level examples, this happens at multiple levels of granularity, all the way up to the project vision itself. It tends to pay back for itself, but you don't need a BDD framework to do BDD.

The conversations in BDD are the important bits, not the tool you use to capture them (I helped write JBehave and still believe this is true). Automating regression tests is also important to cut down the manual effort which rises as the codebase grows, and Cucumber, DSLs and other BDD tools give you this as a very nice by-product which also help you trace, and drive out, the shared understanding.

Edit: I should mention that the reuse of steps is also important, but it doesn't make much difference whether you use a BDD framework or a DSL for that. It does make the difference between a DSL and just procedurally mimicking every user interaction.

Answer 3

It depends what you want to achieve.

Cucumber adds overhead, can be tricky to get used to, takes time to master.

If you want your domain experts to be able to read your tests, you should definitely give it a try.

On the other hand, if developers are the only people reading your tests, you can probably stick to rspec/unit-test/etc. and write your integration tests in those frameworks. However you might achieve more readable high level documentation by using cucumber. See for example rspec 2 core features' descriptions in cucumber.