How to tokenize indents in Coco/R (Python, Boo like indentation)

Question

It there a well known way, how to implement in Coco/R tokenizing indents like in Python/Boo?

Coco/R ignores whitespaces, but I need somehow to generate beginBlock/endBlock tokens, based on next line indent.

Right now, I use preprocessor, which inserts '{', '}', and ';' in input stream, by comparing indents between lines. In Coco/R grammar I use curved braces as beginBlock/endBlock tokens. It works well if input stream has no commens (which could be also nested). As soon as unordered comments coming, intentation comparison logic fails.

Implementing a preprocessor, which traces a comments looks like overenginering to me.

So the question is, is it generally possible to parse indent based grammar with Coco/R? Or should I try something else?

Answer 1

Found a ideal way to do this.

• wrap GetNextToken with method that compares stream positions of the next token with the last one.

• if position.Y is changed, but position.X increased N tabs, inject N virtual INDENT tokens.

• if position.Y is changed, but position.X decreased N tabs, inject N virtual DENDENT tokens.

• if position.Y is changed, but position.X is not, inject virtual SEPARATOR token.

• if position.Y is not changed, return original next token.

• if previous token was a soft break (in python \), ignore logic above.

Answer 2

First of all, Coco/R only ignores blanks (spaces) by default. Tabs are not ignored:

2.3.5 White space
Characters such as blanks, tabulators or end-of-line symbols are usually considered as white space that should be ignored by the scanner. Blanks are ignored by default. If other characters should be ignored as well the user has to specify them in the following way:

WhiteSpaceDecl = "IGNORE" Set.

Example IGNORE '\t' + '\r' + '\n'

I haven't tested this, but my guess is that you should overwrite the default behavior of the Scanner:

Token NextToken() {
    while (ch == ' ' ||
        false
    ) NextCh();

The easiest way to do this is by first modifying the generated code until it works OK and then making the same changes in the frame files (Scanner.frame and Parser.frame) so you won't loose the changes after you regenerate the code.