.NET Test Harness what should it have

Question

We have a software house developing code for us on a project, .NET Web Service (WCF) and we are also paying for a test harness to be built as a separate billable task on a daily rate. I have just joined the company and am reviewing what we are getting from the software house and wanted to know what you guys in industry thought about it?

Basically what we got was a WinForm that called the w/s that had an input area (Web Service Request) to drop our XML a Submit button along with a response area for the result of the Web Response and that's it... Our internal BA has created all the xml request documents so there was no logic put into the harness around this.

Looking on the Net for a definition of a Test Harness I got this: http://en.wikipedia.org/wiki/Test_harness

It states it should have these 3 below things:

• Automate the testing process.
• Execute test suites of test cases.
• Generate associated test reports.

Clearly we have got none of this apart from a partial "Automate the testing process" via a WinForm.

OK, from my development background I would expect someone to Produce a WinForm as a test harness 5 years ago and really should be using some sort of Tooling around this, I explicitly told the Software House I expected some sort of tooling (NUnit,NBUnit, SOAPIU) so we could create a regression test pack for future use. [Didn’t get it but I asked for this after the requirements were signed off as I wasn’t employed then :)]

Would someone be able to clarify with me if my requirement for this is over realistic, I know if I did this, I would use NUnit and TDD and then reuse the test harness as a regression test pack in future?

I am interested to see what the community thought.

Cheers

Updated:

Hey Beniaminus, thanks for the response, glad to see I wasn't going insane, well not yet anyway :) What we have been given is more of a basic "unit test" WinForm I would have used in .NET 1.1 to do "Unit Testing" and not a test harness we are paying for as a Customer.

Granted we have only paid for 1 days developer effort for a test harness but the web service is quite simple and from talking to the BA there is a total of 10 tests (positive and negative) in all.

The WinForm given I think would have taken a max of 2 hours/2 half hours, 30 mins for UI and 1 hour for adding web reference and generating the client side proxy object and calling the service and 30 mins to test it.

I think they could have spun up NUnit created 10 TestFixture tests to read the XML from the file system (created by BA), then called the Web Service from NUnit and parsed the result (pass/fail) by Assertion of the response, well that's what I would have expected anyway :)

Answer 1

I guess, at the most basic level, a test harness is simply a small program that enables you to exercise something that has no UI. However, a basic harness I might whip up to see the results of my web service as I develop it is not the test harness I would expect to receive from something I'm buying.

If I was providing a test harness to someone else, I'd probably do much more to expose a series of test cases, explore edge cases, and allow automation. A test harness like this would therefore document and give examples for how to use the API on service.

The ultimate would be to have a test harness of NUnit tests. Especially if I had already written an NUnit test library, then providing this code to someone else would be free, and would clearly explore and document the API.

So, I don't believe your requirement for automated test harness is reasonable. But without a definition up front, you'll probably rarely get that sort of test harness.