Using ParseKit Tokenization and DelimitedString

Question

I'm trying to use ParseKit's Tokenization with SQL Server syntax. If you take a look at the following example:

CREATE PROCEDURE [dbo].[test]
  @MyId UNIQUEIDENTIFIER
AS
BEGIN
  -- something something
END
GO

I'm using the following to setup comments (and multiline comments):

PKTokenizer *t = [PKTokenizer tokenizerWithString: [inputString substringWithRange: range]];

// SQL does not support // comments.
[t.commentState removeSingleLineStartMarker: @"//"];

// But it does support -- comments.
[t setTokenizerState:t.commentState from: '-' to:'-'];
[t.commentState addSingleLineStartMarker: @"--"];

[t setTokenizerState: t.commentState from: '/' to: '/'];
[t.commentState addMultiLineStartMarker: @"/*" endMarker: @"*/"];

Which sets up support for handling sql comments. Now I'm trying to add support for detecting parameters such as @MyId. Current the tokenizer picks these up as words, but does not include the @ in the string value. My thought was to attempt using a delimitState by doing:

[t.delimitState addStartMarker: @"@" endMarker: @" " allowedCharacterSet: [NSCharacterSet alphanumericCharacterSet]];

Then check for token.isDelimitedString, but it does not seem to pick any up.

Am I going about this wrong? Any suggestions on how I could include the @ in my wordstate?

Answer 1

Developer of ParseKit here.

To match @MyId as a single delimited string token, try:

[t setTokenizerState:t.delimitState from:'@' to:'@'];
[t.delimitState addStartMarker:@"@" endMarker:nil allowedCharacterSet:[NSCharacterSet alphanumericCharacterSet]];