Acceptance Tests and Code Coverage

Question

What is the best way to measure coverage for your Acceptance Tests??

How do you define how much your acceptance tests cover and when they are enough??

Answer 1

When doing acceptance testing, I'd rather look at the functional coverage i.e. how much features (or user stories or use cases) of a given application are tested. And to me, for each feature (or user story or use case), all scenari or possible paths should be tested. In other words, features should be the starting point to write a test plan and the goal should be to cover features, not code. This is a slight difference but an important one: measuring acceptance testing with code-coverage is developer oriented, measuring functional coverage is end-user oriented.

Regarding measurement, Ron Jeffries, one of the founding fathers of Agile, wrote an excellent summary of what he considers a truly valuable metric, RTF or Running Tested Features. You can view the article here. I'm just quoting a small part below:

What is the Point of the Project?

I'm just guessing, but I think the point of most software development projects is software that works, and that has the most features possible per dollar of investment. I call that notion Running Tested [Features], and in fact it can be measured, to a degree.

Imagine the following definition of RTF:

1. The desired software is broken down into named features (requirements, stories) which are part of what it means to deliver the desired system.
2. For each named feature, there are one or more automated acceptance tests which, when they work, will show that the feature in question is implemented.
3. The RTF metric shows, at every moment in the project, how many features are passing all their acceptance tests.

How many customer-defined features are known, through independently-defined testing, to be working? Now there's a metric I could live with.

To report progress, I especially like cumulative flow diagrams. They clearly show when something is going wrong, e.g. (here a burnup style):

_{(source: xprogramming.com)}

For a real life example, have a look at this post or this one (burndown style chart).