How to make ParseKit recognise newline characters?

Question

I'm trying to make ParseKit's Tokenizer recognise newline characters and no other whitespace characters.

Also, the examples show how to make it recognise comments starting with a hash but the functions created seem to apply only to single chars. I'd like to make PK recognise comments that begin with two dashes ('--') and end in a newline.

Answer 1

Developer of ParseKit here.

I'm not sure I understand the question, but if you are looking for the way to make PKTokenizer recognize -- as a single-line comment marker (like in AppleScript), this is it:

PKTokenizer *t = [PKTokenizer tokenizerWithString:str];

// make sure `--` is recognized as a single multi-char token
[t.symbolState add:@"--"];

// designate `--` as a single-line comment marker
[t.commentState addSingleLineStartMaker:@"--"];

// make sure `commentState` handles `-` chars first (to check for comments)
[t setTokenizerState:t.commentState from:'-' to:'-'];

This is explained more fully in the Tokenizer documentation.