xtext: Relation between AST, Metamodel and parse tree

Question

Can someone explain me the relationship between a parse tree, AST and metamodel. I know so far that xtext derive an EMF Ecore metamodel out of the grammar and generate a parser with antlr. But how will it then be parsed: An input goes first trough the lexer and then the parser creates a parse tree out of the parser rules, right? And out of the parse tree, Xtext creates also an AST? For what? And what purpose has the metamodel in this case? I'm a little bit confused of all the definitions.

Answer 1

You are right about the three-step parsing procedure: first the lexer starts with the input stream, then an Antlr-based parse tree is created, finally Xtext generated an EMF-based AST from the parse tree. The first two steps are natural for every parser (generator), the third step needs some explaining. I will start a bit lengthy explanation with some motivation, then I will shortly speak about metamodels and EMF in general.

First of all, the generated parsers do not support identifier resolution (required for handling variables or function calls), these functions needs to be added manually, so a manually coded post-processing step is needed for almost all languages, that needs an extension of the already existing parse tree.

Second, EMF provides a nice, type-safe API for its models, together with a powerful reflective API, that allows the creation of very generic, but useful components that ease the processing of the models (e.g. code generators such as Acceleo or one aspect of Xtend, model transformation tools, such as ATL, ETL, VIATRA2). I cannot tell exactly the difference between the parse tree API of Antlr and EMF, but I worked with the API of the LPG parser generator, and in my opinion, EMF is easier to work with.

Even better, the use of EMF allows the re-use of the rich Xtext functionality together with other EMF-based editors, such as GMF-based graphical editors. See an earlier EclipseCon presentation for the basic idea: TMF meets GMF - Combining Textual and Graphical Modeling.

In general, if we need to extend our parse tree with resolution information, then by re-using an already used paradigm can ease the integration of our language with other tools.

EMF relies on the concept of metamodeling: we have to define the set of elements usable in the models, together with additional constraints, such as information about connectivity. This concept is similar to schema definitions for XML (such as DTD or SML Schema) - we have a uniform way to describe models. Xtext works together with EMF in several ways:

• First of all, based on the grammar, it generates and registers an EMF metamodel that can be used in every EMF-based tooling.
• Then the end result of the parsing process is an EMF model, that can be read and modified using the EMF API - changes are serialized back into the textual form.

I hope, the answer was clear enough. Feel free to ask for more clarifications if needed.