ParseKit not matching my EOL symbols: What am I doing wrong?

Question

The ParseKit adventure continues... and my next hurdle is trying to get newline symbols recognized.

Here's my grammar:

@symbolState = '\n';
@start = textline*;
textline = Any* eol;
eol = '\n';

Here's my test text:

1
2
3
4
5

The text is being read from a UTF-8 text file with Unix format (LF) line endings. I've verified that format both in Xcode (file inspector -> Text Settings) as well as externally, with TextWrangler.

And here's the relevant code:

#import "FileImporterThing.h"
#import <ParseKit/ParseKit.h>

@interface FileImporterThing ()
@property (nonatomic, retain)PKParser* parser;
- (void)parser:(PKParser *)p didMatchTextline:(PKAssembly *)a;
- (void)parser:(PKParser *)p didMatchEol:(PKAssembly *)a;
@end


@implementation FileImporterThing

@synthesize parser = _parser;

-(id)init
{
    if (!(self = [super init])) return nil;

    // Have also tried "textline = Any* '\n';"
    NSString *g = @"@symbolState = '\n'; @start = textline*; textline = Any* eol; eol = '\n';";
    self.parser = [[PKParserFactory factory] parserFromGrammar:g assembler:self];

    return self;
}

- (void)testParse
{
    // read string from UTF-8 file Unix (LF) line endings 
    // (this verified in project->file inspector->Text Settings and externally with TextWrangler)
    NSString *path = [[NSBundle bundleForClass:[self class]] pathForResource:@"LF-test" ofType:@"parsetext"];
/* file contains text:
1
2
3
4
5

*/
    NSString *s = [NSString stringWithContentsOfFile:path encoding:NSUTF8StringEncoding error:nil];

    [self.parser parse:s];
}

- (void)parser:(PKParser *)p didMatchEol:(PKAssembly *)a
{
    NSLog(@"eol found");// stack = %@", [a stack]);
}

- (void)parser:(PKParser *)p didMatchTextline:(PKAssembly *)a
{
    NSLog(@"textline matched");
}

@end

But I'm afraid the paint just isn't drying! Have tried all sorts of variations on the above. I can't for the life of me get the parser to recognise a newline. I can probably work around this, by reading the file-buffer line by line (which might be more performant anyway?) but it would still be nice to have the option of matching '\n'.

Answer 1

Developer of ParseKit here. Two things I can contribute:

---

1.

I tried your example using a text file created in either TextMate or TextWrangler (saved as a .txt file), and everything seemed to work fine. My -parser:didMatchEol: and -parser:didMatchTexline: callbacks were called as expected.

If this is not working for you, maybe try starting with in-memory string input to see if at least that will work:

NSString *s = @"foo bar\nbaz bat\n";
[parser parse:s];

Anyway, here's my code from DebugAppDelegate.m which is working for me with either in-memory input or on-disk input:

- (void)doTestGrammar {
    NSString *g = @"@symbolState = '\n'; @start = textline*; textline = Any* eol; eol = '\n';";
    PKParser *p = [[PKParserFactory factory] parserFromGrammar:g assembler:self];
    //NSString *s = @"foo bar\nbaz bat\n";
    NSString *path = [@"~/Desktop/text.txt" stringByExpandingTildeInPath];
    NSString *s = [NSString stringWithContentsOfFile:path encoding:NSUTF8StringEncoding error:nil];
    [p parse:s];
}

- (void)parser:(PKParser *)p didMatchEol:(PKAssembly *)a {
    NSLog(@"%s %@", __PRETTY_FUNCTION__, a);
}

- (void)parser:(PKParser *)p didMatchTextline:(PKAssembly *)a {
    NSLog(@"%s %@", __PRETTY_FUNCTION__, a);
}

---

2.

I was actually a little surprised that your example worked however, as I thought your use of the Any* production would greedily consume the \n tokens at the end of the line, thus leaving nothing for the eol production to match.

However, as I mentioned, this was not a problem when I ran your example (against ParseKit HEAD of trunk).

Still, for good measure, I would probably recommend altering your textline production to this:

textline = ~eol* eol;

the definition of this production should be read as:

zero or more tokens NOT matched by eol, followed by one token matched by eol

~ is the boolean negation operator in ParseKit grammar syntax.

After making this change to my example code, everything still worked as expected.

---

If you still can't get it working after these two tips, let me know.