Building effective external DSLs
-
01-07-2019 - |
Question
What tools are there for me to build a real, honest to goodness external DSL. And no, I'm not talking about abusing Ruby, Boo, XML or another existing language or syntax, I mean a REAL external DSL -- my own language for my own purposes.
I know that there are a few language workbenches being developed and I've heard about things like "Irony" for .NET. And, of course, there's ANTLR, Lex/Yaac, etc but I'm afraid those are too complicated for what I'm trying to do.
Please talk about a DSL builder tool you may have used or heard about and your impressions on how it helps and what its downsides are.
Solution
I've written DSLs in Boo, Irony.NET and a toolkit called Grammatica. You say that a parser-generator is too complicated, but you may be being too hasty in your judgment, in fact they are quite simple to use once you get over a small learning curve, and open up a vast world of possibility that easily overrides the effort. I found learning the notation required to write grammars for most parser generators somewhat similar to learning Regular Expressions - you have to bend your mind just slightly to let them in, but the rewards are significant.
My opinion is this: If your target language is simple enough that it could be handled by a dumbed down visual designer, then writing a grammar for it using a parser generator should be quite easy.
If your target DSL is complicated enough that you'll need to break a sweat writing a grammar, then the dumbed down visual tool won't cut the mustard anyway and you'll end up having to learn to write a grammar anyway.
I agree in the long term about internal vs external DSL's, though. I wrote an internal DSL in Boo and had to modify my DSL syntax to make it work, and it always felt like a hack. The same grammar using Irony.NET or ANTLR would have been just as easy to accomplish with more flexibility.
I have a blog post discussing some options. The post is centered around writing a DSL for runtime expression evaluation, but the tools are all the same.
My experience with Irony.NET has been all positive, and there are several reference language implemented using it, which is a good place to start. If your language is simple, it is absolutely not complicated to get up and running. There is also a library on CodeProject called TinyParser - this one is really interesting, because it generates the parser as pure source code, which means your final product is completely free of any third party reference. I haven't used it myself, though.
OTHER TIPS
If you're looking into writing stand-alone DSLs, then you're looking into building compilers--no way around it. Compiler construction is essential programming knowledge, and it's really not as difficult as commonly thought. Steve Yegge's Righ Programmer Food summarizes the value of knowing how to build compilers quite nicely.
There are plenty of ways to get started. I recommend checking out the 2 papers mentioned in the article: Want to write a compiler? Just read these Two papers. The first one, Let's build a compiler, is very accessible. It uses Turbo Pascal as an implementation language, but you can easily implement it in any other language--the source code is very clear. Pascal is a simple language.
Once you get a good feel for how things work and the terminology involved, I recommend delving into something like ANTLR. ANTLR has a nice IDE, ANTLRWorks, that comes with an interpreter and a debugger. It also produces really really good visualizations of your grammars on the fly. I found it invaluable in learning.
ANTLR has several good tutorials, although they might be a bit overwhelming at first. This one is nice, although it's against ANTLR 2.0, so you might run into incompatibilities with a more recent version (currently the latest is 3.1).
Finally, there's another approach to DSLs: The Lisp approach. Given Lisp's syntax-less nature (your code is basically abstract syntax trees), you can shape endless languages out of it, provided you get used to the parentheses :).
If you do go with that approach, you want to use an embeddable Lisp. Under Java, you have Clojure, a Lisp dialect that interoperates flawlessly with JVM and its libraries. I haven't used it personally, but it looks good. For Scheme, there's GNU Guile, which is licensed under LGPL. For Common Lisp, there's ECL, also under the LGPL. Both use a C interface for interoperability, so you can pretty much embed them into any other language. ECL is unique among Lisps in that each Lisp function is implemented as a C function, so you can write Lisp code in C if you want to (say, inside your own extensions methods--you can create C functions that operate on Lisp objects, and then call them from Lisp). I've been using ECL for a side-project of mine for a while, and I like it. The maintainer is quite active and responsive.
You should really check out Ragel. It's a framework to embed state machines in your regular source code. Ragel supports C, C++, Objective-C, D, Java and Ruby.
Ragel's great for writing file and protocol parsers as well as stepping through external DSL stuff. Chiefly because it allows you to execute any kind of code on state transitions and such.
A couple of notable projects that use Ragel are, Mongrel, a great ruby web server. And Hpricot, a ruby based html-parser, sort of inspired by jQuery.
Another great feature of Ragel is how it can generate graphviz-based charts that visualize your state machines. Below is an example taken from Zed Shaw's article on ragel state charts.
Xtext was built for this.
From the website:
Xtext is a framework for development of programming languages and domain specific languages.
It covers all aspects of a complete language infrastructure, from parsers, over linker, compiler or interpreter to fully-blown top-notch Eclipse IDE integration. It comes with good defaults for all these aspects and at the same time every single aspect can be tailored to your needs.
I've been using Irony with good results. The great part about irony is that you can easily include it in whatever runtime you'll be using the DSL for. I'm creating an external DSL that I populate into a semantic model written in C# so irony is great. Then I use the semantic model to generate code with StringTemplate.
If you are planning to implement an external DSLs , Spoofax ( http://strategoxt.org/Spoofax )is a nice Language Workbench to do this. It is a parser-based textual Langauge Workbench that leverage several state-of-art technology such as SDF , Stratego. Besides the DSL implemenation , you could get a very rich editor services such as, code completion , outline view , intellisense etc. It has been used to build several languages e.g. http://mobl-lang.org/. Check this out to get the idea about the provided support.
Spoofax project comes with a out-of-the box nice sample DSL implementation and a java code generator. It may work as a starting point to get started with the tools.
Following tutorial details about the usage this langauge workbench : http://strategoxt.org/Spoofax/Tour.
Hope it helps!
For serious external DSLs, you can't avoid the parsing problem; ANTLR is the least of what you need. What you want to check is program transformation systems, which can be used to map arbitrary DSL syntax into target languages like Java.