Your design might not be ideal, but it's difficult to know without looking at the code.
Something to ask yourself is the Parser hiding too much information, and as a consequence doing too much.
I strongly suggest you to watch this presentation from Michael Feather, in which (interestingly) he discusses how to improve the design of a parser using tests, and it looks like the problem he solves is similar to what you're trying to solve. Michael Feathers - the deep synergy between testability and good design