End to end testing framework recommendations

Question

I am in a new project that is looking to include an end to end testing framework. We want something flexible, I've used Fitnesse before and I think we need something similar to it.

We are also using Hudson CI and are looking for something that would integrate easily with it.

Is there a clear winner?

UPDATED: The system has many components, some of them are web services running on tomcat, there are a couple of NoSQL databases too, but no UI testing is required for the moment.

Please add a comment if further clarification on the project details is needed.

Answer 1

The robot framework is a good keyword driven testing framework that we use for end to end integration testing. http://code.google.com/p/robotframework/

There is a plugin for jenkins (a fork of hudson) that is very useful. It reportedly works with hudson as well.

Answer 2

Great to read you have decided start using Robot Framework. Its Hudson plugin already has recently got much better (trend charts, available from Hudson directly, ...) and moved to new place: http://wiki.hudson-ci.org/display/HUDSON/Robot+Framework+Plugin

Answer 3

Can you be more specific, when you say, you want to have an end-to-end testing framework? What kind of application are you developing (standalone, web based, web service, ...)? What language do you program in?

I would also look for previous knowledge in your team or around your team (e.g. other teams). It might be sufficient, to use TestNG or JUnit (I have seen an old paper on UI testing using JUnit). Almost all test frameworks should be fine for Hudson, as long as they can be started in a headless/scripted mode and if they provide the results in a format that Hudson understands (in the worst case HTML is sufficent, though Hudson will only be able to display the pages). So if you have IBM Monitoring tools available, you might want to look at Rational Functional Tester or Rational Performance Tester.

FitNesse is a good tool for functional testing. The business contacts can create the test cases by themselfs (developers have to provide some connector code), which will create more visibility and removes some work from the test engineers. Drawback is, that you can't really do UI testing. If you need a open source plattform for UI, have a look at seam. Even though I like the OSS philosophi, you might need longer ramp up time (higher costs) and possibly more maintainace time for OSS (not true for all OSS). In any case, check how easy it is to get support for your framework. You might even consider paid support for your framework (regardless if open or closed source). Also don't forget, even though you can fix OSS source yourself, you might not have the expertise or time for fixing bugs or adding features.

So give us more info and we can give you better advices.

Answer 4

My experience with jBehave is very good. Recommend.

It's based on BDD - Behavior Driven Development.