Test plans and how best to write them

Question

We're trying to figure out the best way to write tests in our test plan. Specifically, when writing a test that is meant to be used by anyone including QA staff, should the steps in the test be very specific or more broad giving the tester more leeway in how the task can be accomplished. As a very simple example, if you're testing opening a document in word processing document, should the test read:

1. Using the mouse, open the file menu
2. Choose "Open File..." in the file menu
3. In the open file dialog that appears, navigate to x and double-click the document called y

OR

1. Bring up the file open dialog
2. Open the file y

Now I realize one answer is probably going to be "it depends on what you're trying to test" but I'm trying to answer a broader question here: If the test steps are too specific do we risk a) making the testing process to laborious and tedious and more importantly b) do we risk missing something because we wrote down too specific a path to achieve a goal. Alternatively, if we make it broad do we depend too much on the whims of the tester at the time and lose crucial testing of paths that are more common to customers/clients?

Answer 1

My first question would be - why isn't your QA department writing the test plans? Typically, software developers provide QA with a functional spec of how the software is supposed to work and then QA creates test plans based on that.

With that said, I would suggest being very specific with the steps since you are detailing how things are supposed to work. It is then the job of the tester to make sure that your specific steps get the desired results and it is also their job to deviate from the plan and try to break things.

If there are multiple ways to achieve a goal, you need to describe each path to get there.

Answer 2

I'm not a tester but in my opinion it is vital to document the "UI route" that the test must take to avoid any confusion.

I have worked on countless defects that I could not reproduce simply because I was not accessing the function from the same UI path as the tester was. e.g. Right Click menu vs Toolbar or functions that can be carried out from various dialogs, etc. etc.

Answer 3

It sounds like your QA staff is really QC (Quality Control) if they are not responsible for writing tests. If they actually are responsible for writing tests, your tests will be helpful, but specifications that are clear would be a better source for them to write the tests themselves. Even better would be to have them as part of the review process for the specs so that they can ask for details that will allow them to write tests.

If you really are in a position where you are writing tests for other people, there are some considerations. You will want a painful level of detail if :

• the people running the tests are not able to come ask you questions
• the people running the tests are not familiar with your product

You can avoid some details if these are not true. However, it still depends :)

This all being said, what you will have written is not what is generally considered a 'test plan'. A Test Plan describes what types of tests will be executed (functional, regression, security, etc.), what features are to be tested, maybe even what tests are to be written, who will be doing the testing, when groups of tests are scheduled and other planning type activities.

What you describe above is simply a set of tests.

Answer 4

The first is feature testing. Test with detailed steps containing UI route as there is possibly more routes than one to the destination. Test all routes. The latter sounds more like usability testing. It should be done too but not only by your testers but also by external people.

Answer 5

Let's distinguish Test Plan and Test Suites :)

Test Suite is set of tests itself

Test Plan is [part of] Test Suite + available resources (people, hardware, time, ...).

It's OK to have both variants (detailed and "rough") described in your test documentation, just place detailed and "rough" tests to different documents and prioritize these documents.

Then, when you have enough time to test product completely, you take all documents of, say, category A and test product in accordance with these documents. If you have no time, but need quick conclusion about quality, you take category B documents and pass checks described there.

good side: you can select how to test product

bad side: you need "duplicate" documents

Answer 6

It's perfectly fine to want exact, detailed, repro steps when someone finds a problem. But if you write your test plans that way, you will risk the following problems:

1) Inattentional blindness - I have watched people executing a detailed procedural test script, dutifully walking through and recording every step meticulously, and TOTALLY MISSING the glaring error right in front of them. Because it "wasn't in the script". Their attention was so focused on all those finicky test steps that they literally could not see the problems in front of them.

2) You will miss ALL those bugs which are just one step off your highly detailed, very specific path. When customers get your product, they won't follow the detailed test plan. They will navigate around your app in a variety of ways. They will change their minds. They will have names longer, or shorter, than you thought probable or possible. They will get halfway through a transaction and abandon it. They will wander. They won't stick to one path. And every time someone repeats the test, they will miss those bugs again.

3) You will spend an incredibly long time trying to get "anyone can follow this" test scripts written. Believe me, I spent years trying to perfect this, and it's just not humanly possible. Worse still, the amount of time you waste trying to do this could be spent much more profitably in some other way, so your product is worse off.

4) You'll end up with a ton of repetition, and it will be hard to tell what the point of your test is without reading the whole thing. It won't be easy to scan quickly through the tests to see what use cases you may have missed.

Keep your test plans broad and allow the people doing the testing to exercise their judgement. If you have information about specific usage scenarios that must be tested, or about how the target user group will want to operate, then give this to the testers as well along with the test plans - perhaps in the form of user personas, perhaps just in the form of use cases. If you need specific things ticked off, consider using a checklist. (For more information, see Cem Kaner's excellent presentation www.kaner.com/pdfs/ValueOfChecklists.pdf).

Alternatively, write your test plans as short exploratory charters. You could, for example, give guidance such as: "Callcentre users will be using workstations with no mouse attached. Explore the process of raising a ticket on behalf of a customer, ensuring that it's possible to complete all activities using keyboard shortcuts only." This is far more likely to result in your testers finding defects than saying "Tab into field 1. Enter "Complaint about line quality". Tab into field 2. Select "Phone call" from the dropdown menu. Tab into .... field 68."

Answer 7

theres pros and cons to treating your tester like they have no knowledge of the system or computers in general.

if you spell things out in minute detail (e.g. "from the file menu, select 'Open'...") than the benefit is you can use contractors that arent familiar with your system. but it takes you longer to write like this

if you skip a lot of detail (e.g. "open a document file..."), than whoever uses your test plan is more likely to get stuck, and than interrupt you for clarification. but it is much faster to write

it can be a false economy to think its faster if you do the brisk version, if you end up just spending extra time explaining the system to the qa person

i have an article where i go into more depth on this topic: Writing a System Test Plan

in this article, i favour the more detailed approach. but i have been developing a 'mid-point' between these two approaches lately (called a FEATURE test plan), but im not at a point where its mature enough to share yet

-- LM