I actually managed to find a solution on my own. I defined my own tokenizer function which decides based on the BufferLocalStore
state which tokenizer to call.
let mytokenizer (lexbuf : LexBuffer<char>) =
if lexbuf.BufferLocalStore.["state"].Equals("core") then FCLexer.core lexbuf
else FCLexer.tokenize lexbuf
let aString (x : string) =
let lexbuf = LexBuffer<_>.FromString x
lexbuf.BufferLocalStore.["state"] <- "fc"
let y = try (FCParser.PROG mytokenizer) lexbuf
...
And I modified my fslex
input file slightly:
rule tokenize = parse
| "core" { lexbuf.BufferLocalStore.["state"] <- "core"; core lexbuf }
...
Amazing how simply asking the question can lead you to the solution, and I hope this helps someone besides me :)