What is the most mature BDD Framework for .NET? [closed]

Question

We have been using BDD - Behaviour Driven Development (from Dan North's perspective) as a mechanism to record user acceptance tests and drive development on a couple of projects, with decent success. To date though we have not actually automated the tests themselves.

I am now looking in to automating the tests, but I am not sure which behaviour framework to back. So far NBehave seems to be the forerunner - but are there any others I should be looking at? Is there a clear 'winner' at the moment?

Answer 1

The Quick Answer

One very important point to bring up is that there are two flavors of Behavior Driven Development. The two flavors are xBehave and xSpec.

xBehave BDD: SpecFlow

SpecFlow (very similar to cucumber from the Ruby stack) is excellent in facilitating xBehave BDD tests as Acceptance Criteria. It does not however provide a good way to write behavioral tests on a unit level. There are a few other xBehave testing frameworks, but SpecFlow has gotten a lot of traction.

xSpec BDD: MSpec

Objectively speaking. Given the context specifications frameworks available, MSpec has been around the longest and is the most widely used context/specification framework in the .Net community.

The other xSpec BDD framework: NSpec

I personally would recommend NSpec (inspired directly by RSpec for Ruby). Full disclosure, I am one of the authors of NSpec. You can accomplish BDD by simply using NUnit or MSTest...but they kinda fall short (it's really hard to build up contexts incrementally). MSpec is also an option and is the most mature context/specification framework for .Net. But, there are just some things that are simpler in NSpec.

The Long Answer

The two flavors of BDD primarily exist because of the orthogonal benefits they provide.

Pros and Cons of xBehave (GWT Syntax)

Pros

• helps facilitate a conversations with the business through a common dialect called (eg. Given ...., And Given ...., When ......, And When ..... , Then ...., And Then)
• the common dialect can then be mapped to executable code which proves to the business that you actually finished what you said you'd finish
• the dialect is constricting, so the business has to disambiguate requirements and make it fit into the sentences.

Cons

• While the xBehave approach is good for driving high level Acceptance Criteria, the cycles needed to map English to executable code via attributes makes it infeasible for driving out a domain at the unit level.
• Mapping the common dialect to tests is PAINFUL (ramp up on your regex). Each sentence the business creates must be mapped to an executable method via attributes.
• The common dialect must be tightly controlled so that managing the mapping doesn't get out of hand. Any time you change a sentence, you have to find method that directly relates to that sentence and fix the regex matching.

Pros and Cons of xSpec (Context/Specification)

Pros

• Allows the developer to build up context incrementally. A context can be set up for a test and some assertions can be performed against that context. You can then specify more context (building upon the context that already exists) and then specify more tests.
• No constricting language. Developers can be more expressive about how a certain part of a system behaves.
• No mapping needed between English and a common dialect (because there isn't one).

Cons

• Not as approachable by the business. Let's face it, the business don't like to disambiguate what they want. If we gave them a context based approach to BDD then the sentence would just read "Just make it work".
• Everything is in the code. The context documentation is intertwined within the code (that's why we don't have to worry about mapping english to code)
• Not as readable given a less restrictive verbiage.

Samples

The Bowling Kata is a pretty good example.

SpecFlow Sample

Here is what the specification would look like in SpecFlow (again, this is great as an acceptance test, because it communicates directly with the business):

Feature File

The feature file is the common dialect for the test.

Feature: Score Calculation 
  In order to know my performance
  As a player
  I want the system to calculate my total score

Scenario: Gutter game
  Given a new bowling game
  When all of my balls are landing in the gutter
  Then my total score should be 0

Scenario: Single Pin
  Given a new bowling game
  When I've hit exactly 1 pin
  Then my total score should be 1

Step Definition File

The step definition file is the actual execution of the test, this file includes the mappings for SpecFlow

[Binding]
public class BowlingSteps
{
    private Game _game;

    [Given(@"a new bowling game")]
    public void GivenANewBowlingGame()
    {
        _game = new Game();
    }

    [When(@"all of my balls are landing in the gutter")]
    public void WhenAllOfMyBallsAreLandingInTheGutter()
    {
        _game.Frames = "00000000000000000000";
    }

    [When(@"I've hit exactly 1 pin")]
    public void When1PinIsHit()
    {
        _game.Frames = "10000000000000000000";
    }

    [Then(@"my total score should be (\d+)")]
    public void ThenMyTotalScoreShouldBe(int score)
    {
        Assert.AreEqual(score, _game.Score);
    }
}

MSpec Sample, xSpec, Context/Specification

public class describe_BowlingKata
{
    public static Game game;

    public class when_all_balls_land_in_the_gutter : describe_BowlingKata
    {
        Establish ctx = () => game = new Game();

        Because of = () => game.Frames = "00000000000000000000";

        It should_have_a_score_of_0 = () => game.Score.ShouldBe(0);
    }

    public class when_a_single_pin_is_hit : describe_BowlingKata
    {
        Establish ctx = () => game = new Game();

        Because of = () => game.Frames = "10000000000000000000";

        It should_have_a_score_of_1 = () => game.Score.ShouldBe(1);
    }
}

NSpec Sample, xSpec, Context/Specification

Here is a NSpec example of the same bowling kata:

class describe_BowlingGame : nspec
{
    Game game;

    void before_each()
    {
        game = new Game();
    }

    void when_all_my_balls_land_in_the_gutter()
    {
        before = () => game.Frames = "00000000000000000000";

        it["should have a score of 0"] = () => game.Score.should_be(0);
    }

    void when_a_single_pin_is_it()
    { 
        before = () => game.Frames = "10000000000000000000";

        it["should have a score of 1"] = () => game.Score.should_be(1);
    }
}

As you do more and more BDD, you'll find that both the xBehave and xSpec flavors of BDD are needed. xBehave is more suited for Acceptance Tests, xSpec is more suited for unit tests and domain driven design.

MSpec vs NSpec

Objective metrics like age and stability should be a factor, and I would encourage everyone to take that into consideration. But please also take into consideration that newer frameworks may provide a more succinct api, better usage of language constructs and build on lessons learned for past frameworks. MSpec provides constructs of Given, Because, It and Cleanup..but they come at a cost: static initialization for all members, class explosion, and it's syntactically rigid because of its unique use of delegates. You'll find that the simplest MSpec tests are simpler in NSpec. Here is a more complex test suite written both in MSpec and NSpec.

A Comparison of xUnit, MSpec and NSpec: https://gist.github.com/amirrajan/6701522

Relevant Links

RSpec vs Cucumber (RSpec stories)

BDD with Cucumber and rspec - when is this redundant?

Answer 2

Check out SpecFlow.

It is a tool inspired by Cucumber that aims at providing a pragmatic and frictionless approach to Acceptance Test Driven Development and Behavior Driven Development for .NET projects today.

VisualStudio integration seems especially promising.

Answer 3

StoryQ looks like a nice alternative to NBehave with its Fluent interface. I would definitely check it out.

Answer 4

I dont think there is a 'winner' really. Other frameworks include SpecUnit.NET project and MSpec is also showing promise with the beginnings of a Gallio adapter. Technically since IronRuby is on the horizon, rSpec may be an option for those prepared to go bleeding edge. NBehave + NSpec might be the eldest framework with the best automation, though I found myself fighting against the overly verbose syntax.

I would check them all out and pick the framework that suits your projects style best. They're all OSS, so its hard to lose, the real benefit is simply in moving to BDD.

Answer 5

I personally would recommend BDDfy which is great in my opinion! It's open source, supports convention and fluent based scenario description, covers great both acceptance and unit tests. You can find it on GitHub.

Answer 6

Robot Framework can also be used with IronPython to do ATDD or BDD in .Net. I think the expression capabilities of Robot Framework are better than eg. SpecFlow's or NSpec's. It doesn't force you to use a certain syntax, but instead uses a keyword driven approach. If you are testing web applications, it has Selenium2Library that provides bindings to Selenium WebDriver.

Answer 7

There's also Specter, which defines a DSL in Boo to make it all more natural.

Answer 8

I'd generally go in favour of NBehave, combined with MbUnit and whichever DI/mocking frameworks you need. It's a fair comment by Jim Burger that NBehave is very verbose, and I find myself using cut-and-paste at times. Still, it works great - I'm using Gallio's ReSharper plug-in, so I just get an extra window showing. Haven't tried it with ccnet yet, though.

Answer 9

Check this blog post and its comments for inspiring ideas: http://haacked.com/archive/2008/08/23/introducing-subspec.aspx/

Answer 10

Concordion.NET does not only support BDD but also ATDD: http://assertselenium.com/2012/11/05/difference-between-tdd-bdd-atdd/ Specifications are written in plain English using HTML. IMHO this is one of the benefits of Concordion.NET. The HTML documents can be organized into a navigable structure to build up a living documentation system. And, since the tests run against the system, you can be confident the documentation is always up-to-date.

Answer 11

I'm starting on my first outing in BDD with a small application with my team. The tools we are choosing to do the job are: Specflow with Selenium Webdriver for xBehave stories and MSpec with Machine.Fakes.Moq for an automocking container for our xSpec unit tests. With Resharper to be both our story runner and specifications runner due to the integration supported by Specflow and MSpec. Having native integration into visual studio with R# is a key feature for us.

Since our design is all MVC3 we will also be applying the orchestrator separation pattern to our MVC controllers. This will allow us to write specifications directly against the orchestrator. Then for us to write stories directly against our application usage.

Answer 12

Since I am now dealing with BDD for system tests for safety critical applications, I can only share my experience that you should not underestimate the power of "tests written in a natural language" instead of "code".

We always focused to offer a feature language, that anybody can understand without any technical background or experience in programming (see the specflow example above) and we did well doing so. Beside the fact that I never explained the syntax to anybody, everybody immediately understood the meaning of the test, developer, manager, and even tester.

I would in any way avoid a test written in a programming language like the MSpec examples above. If I show up with a test like this in the office of a manager, he will kick me out. But he is interested in reading Gherkin-Syntax-based tests. The more guys are able to read and modify the tests, the better.

Last but not least those tests are portable to any other programming language, any other plattform, any other test automation tool without any changes.

Again, the answer is one more time:

Don't care about the tool and its features itself, choose a tool that allows you to easily switch to another tool at any time without loosing information. Tools come and go, my tests should last longer. :-)

I can recommend using SpecFlow. You have full access to the complete .Net Library and all of its features, your tests stay portable. This might give you an advantage over totally portable solutions like the Robot Framework, if you don't mind portability. However, you can always improve stability and portability of a system by using different tools for development and testing. So testing a .Net software with a python based BDD approach might be an good (or even the better) idea in some cases.

However, SpecFlow showed up being stable and bullet-proof in every day testing, including nightly build tests, etc. in medium sized projects. Furthermore, it integrates well into the Microsoft Unit Test Environment.

Answer 13

Also check out UBADDAS, specific to UAT found at

https://github.com/KernowCode/UBADDAS

with an explanation here

http://kernowcode.wordpress.com/ (in June 2014)

You can write a test like this

[Test]
public void IWantToRegisterANewUser()
{
  var user = new User();
  ICustomer customer = new Customer();

  SoThat(MyBusinessValue.IncreaseCustomerBase)
    .As(user)
    .Given(customer.Register)
    .When(customer.Confirm_Registration)
    .Then(customer.Login);
}

and produces this

I want to register a new user
  so that Increase customer base
       as user
    given Register customer
     when Confirm customer registration
     then Login customer