I don't think there really is a problem, because tokens are 'greedily' matched in the order they've been added to the token definitions (for a specific lexer state).
So, given
this->self.add
("{STATUSCODE}", T_STATUSCODE)
("{SP}", T_SP)
("{REASONPHRASE}", T_REASONPHRASE)
;
T_STATUSCODE will always match before T_REASONPHRASE (if there is an ambiguity at all).
About using separate Lexer states, here's an excerpt of a tokenizer I once had in a toy project:
this->self = fileheader [ lex::_state = "GT" ];
this->self("GT") =
gametype_label |
gametype_63000 | gametype_63001 | gametype_63002 |
gametype_63003 | gametype_63004 | gametype_63005 |
gametype_63006 |
gametype_eol [ lex::_state = "ML" ];
this->self("ML") = mvnumber [ lex::_state = "MV" ];
this->self("MV") = piece | field | op | check | CASTLEK | CASTLEQ
| promotion
| Checkmate | Stalemate | EnPassant
| eol [ lex::_state = "ML" ]
| space [ lex::_pass = lex::pass_flags::pass_ignore ];
(The purpose would be relatively clear if you read GT
as gametype, ML
: move line and MV
: move; Note the presence of eol
and gametype_eol
here: Lex disallows adding the same token to different states)