SimpleTest WebTestCases affecting each other, not deterministic output

Question

I am developing some project with CodeIgniter and write unit tests and web tests in SimpleTest. I've noticed that my tests are not deterministic, i.e. they produce different outputs in time. I mean the test cases that should be strictly deterministic, not relying on random variables etc.

The tests look like affecting each other. Quite often, when everything goes okay, I have let's say 100 passed tests, but when I write a new test method that fails, then several other tests also do fail. But often after correcting the problem in my failing test case and re-running whole test suite 2-3 times whole suite gives a pass again.

This happens with WebTestCases generally.

Do you have any idea what could be the problem?

I do not modify any class variables that are shared etc.

I've glance at the code of SimpleTest (more or less, it's big to analyze whole flow quickly) and it looks like the instance of browser is re-created before launching different tests.

The thing that is the strangest is that after re-running, some errors disappear, and finally, all of them. Is there some caching involved in this?

I'll be grateful for hints as there is really not much documentation / blog entries / forum posts about SimpleTest in the web, except its API on the website.

Answer 1

(edit: moved the answer as a separate post)

Huh, I made quite thorough investigation and it seems that there is a bug in SimpleTest library.

They use fsockopen for opening connection, then send request via fwrite, and then incorrectly fetch response from socket. What I mean: it can happen that we read 0 bytes from socket, but we're not done, as we falsely assume, cause the server can be busy, and send data later, while we prematurely ended reading. That way, we haven't read whole response and we do tests against only partial response, causing it to fail.

Answer 2

Things it might be:

• Caching - are you caching bad results somewhere in the chain?
• Misunderstanding - Are you sure you are testing the right things?
• Bad Data - If you are testing this on top of a database, and the failure corrupted the data in the database, you might see results like you mention.