TDD on an already started project

Question

I am currently implementing my own programming language. Until now I have written:

• An Error class for errors (to be thrown) encountered while processing the input source code;

• Some SyntaxError functions (each one with a different tag, passed with a template, like SyntaxError<malformed_number>() or SyntaxError<unexpected_symbol>) which help constructing Error objects with different error messages (for now I only got this, but I will soon have FunctionErrors, IndexErrors and so on);

• A Token class which contains the informations about each token (line in the string at which it was when it was encountered, the type which can be string, number... things like that);

• Finally, a lexer function which takes as imput a string of source code and returns a std::list<Token*> object. Internally, it uses a few helper functions (buildname, builsymbol, skipcomment...) to help mantain the code organized.

I didn't follow any particular programming technique will coding this, but I now feel the urge to follow those that go under the Test Developement Programming name.

But I have a few concerns:

• First, what do I have to test in the above code? Of course the lexer function should be tested deeply, and maybe also the helper functions it uses (I guess), but what about the other things?

• Second, how do I test functions with a non-trivial output? Online I see a lot of examples comparing simple numbers or strings, but how does one write a test for a function which returns a list of pointers to objects whitout having to write too much for a single test case?

Answer 1

TDD is less a testing but more a coding technique. The question you need to ask here is not

"shall I test this class or not"

but

"I want to implement a small change in my code base, how can I write a test which is red before the change, and becomes green after the change"

For example, a small change might be the introduction of a new "reserved word" in your programming language. You write a test for your lexer to return a special token for this - but since you did not implement the feature, the test will fail. Then you implement the feature - test goes green. Next you refactor the code, run the test again, it gets red because you made an error. You fix that error, test goes green. Next feature: using the new reserved word in a specifically wrong manner shall return a SyntaxError exception - write a new test for this - ... - I guess you got the idea.

As you see, it does not really matter for applying TDD if there is an existing code base, or not - TDD is about verifying changes to a code base, not about testing the whole code base in each and every way.

Of course, if a code base was written without "TDD testability" in mind, it can become hard to write tests for certain changes. So if you really want to switch to TDD, it might be easier to do this early.