Amend JSON - based grammar to allow for trailing comma

Question

So Doctrine annotations are based on JSON but as PHP arrays allow for dangling commas Drupal people are much used to this and to be honest, it's much easier to edit

(
  "foo" => "bar",
  "foo1" => "bar1",
)

so that the rows can be switched, another row added etc than if that comma after bar1 were missing.

So I am trying to patch Doctrine to allow for this. The maintainer claims

you are mixing grammar rules by checking a token that is not part of "Values", but its parent, "Annotation".

And he is right. So, we are looking for a better grammar.

Annotation     ::= "@" AnnotationName ["(" [Values] ")"]
Values ::= Array | Value {"," Value}*

How would you allow for a trailing comma in Values?

Answer 1

The modified grammar is not LL(1) anymore. As a consequence the simple recursive descent parser implemented by Doctrine requires backtracking.

In this case, you need to make sure that ::Value() can backtrack, ie. trigger an exception and reset the parsing to the position it was in before entering ::Value(). You catch this exception and just ignore it in ::Values() as soon as you already have one valid value.

That requires to be able to seek in the lexer, and I don't know how practical it is. AbstractLexer is quite a bit weird.

---

Edit: The easiest way forward would be to keep the grammar in a LL(1) form. One way of doing that is to move the parenthesis inside Values:

Values ::= "(" [Array | Value {"," Value}* [","] ] ")"

Answer 2

To add an optional comma at the end we can use the [] notation:

Annotation     ::= "@" AnnotationName ["(" [Values] ")"]
Values ::= Array | Value {"," Value}* [","]