Can I use Scala Macros to internalise an external DSL?

Question

I would like to implement an external DSL such as SQL in Scala using Macros. I have already seen papers on how to implement internal DSLs with Scala. Also, I've recently written an article about how this can be done in Java, myself.

Now, internal DSLs always feel a bit clumsy as they have to be implemented and used in the host language (e.g. Scala) and adhere to the host language's syntax constraints. That's why I'm hoping that Scala Macros will allow to internalise an external DSL without any such constraints. However, I don't fully understand Scala Macros and how far I can go with them. I've seen that SLICK and also a much less-known library called sqltyped have started using Macros, but SLICK uses a "Scalaesque" syntax for querying, which isn't really SQL, whereas sqltyped uses Macros to parse SQL strings (which can be done without Macros, too). Also, the various examples given on the Scala website are too trivial for what I'm trying to do

My question is:

Given an example external DSL defined as some BNF grammar like this:

MyGrammar ::= ( 
  'SOME-KEYWORD' 'OPTION'?
    (
      ( 'CHOICE-1' 'ARG-1'+ )
    | ( 'CHOICE-2' 'ARG-2'  )
    )
)

Can I implement the above grammar using Scala Macros to allow for client programs like this? Or are Scala Macros not powerful enough to implement such a DSL?

// This function would take a Scala compile-checked argument and produce an AST
// of some sort, that I can further process
def evaluate(args: MyGrammar): MyGrammarEvaluated = ...

// These expressions produce a valid result, as the argument is valid according
// to my grammar
val result1 = evaluate(SOME-KEYWORD CHOICE-1 ARG-1 ARG-1)
val result2 = evaluate(SOME-KEYWORD CHOICE-2 ARG-2)
val result3 = evaluate(SOME-KEYWORD OPTION CHOICE-1 ARG-1 ARG-1)
val result4 = evaluate(SOME-KEYWORD OPTION CHOICE-2 ARG-2)

// These expressions produce a compilation error, as the argument is invalid
// according to my grammar
val result5 = evaluate(SOME-KEYWORD CHOICE-1)
val result6 = evaluate(SOME-KEYWORD CHOICE-2 ARG-2 ARG-2)

Note, I'm not interested in solutions that parse strings, the way sqltyped does

Answer 1

It's been some time since this question was answered by paradigmatic, but I've just stumbled upon it and thought it's worth extending.

An internalized DSL must indeed be valid Scala code with all the names defined before macro expansion, however one can overcome this restriction with a carefully designed syntax and Dynamics.

Let's say we wanted to create a simple, silly DSL allowing us to introduce people in a classy way. It might look like this:

people {
  introduce John please
  introduce Frank and Lilly please
}

We would like to translate (as part of compilation) the above code to an object (of a class derived for example from class People) containing definitions of fields of type Person for every introduced person - something like this:

new People {
    val john: Person = new Person("John")
    val frank: Person = new Person("Frank")
    val lilly: Person = new Person("Lilly")
}

To make it possible we need to define some artificial objects and classes having two purposes: defining grammar (somewhat...) and tricking the compiler into accepting undefined names (like John or Lilly).

import scala.language.dynamics

trait AllowedAfterName

object and extends Dynamic with AllowedAfterName {
  def applyDynamic(personName: String)(arg: AllowedAfterName): AllowedAfterName = this
}

object please extends AllowedAfterName

object introduce extends Dynamic {
  def applyDynamic(personName: String)(arg: AllowedAfterName): and.type = and
}

These dummy definitions make our DSL code legal - the compiler translates it to the below code before proceeding to macro expansion:

people {
    introduce.applyDynamic("John")(please)
    introduce.applyDynamic("Frank")(and).applyDynamic("Lilly")(please)
}

Do we need this ugly and seemingly redundant please? One could probably come up with a nicer syntax, for example using Scala's postfix operator notation (language.postfixOps), but that gets tricky due to semicolon inference (you can try it yourself in the REPL console or IntelliJ's Scala Worksheet). It's easiest to just interlace keywords with undefined names.

As we've made the syntax legal, we can process the block with a macro:

def people[A](block: A): People = macro Macros.impl[A]

class Macros(val c: whitebox.Context) {
  import c.universe._

  def impl[A](block: c.Tree) = {
    val introductions = block.children

    def getNames(t: c.Tree): List[String] = t match {
      case q"applyDynamic($name)(and).$rest" =>
        name :: getNames(q"$rest")
      case q"applyDynamic($name)(please)" =>
        List(name)
    }

    val names = introductions flatMap getNames

    val defs = names map { n =>
      val varName = TermName(n.toLowerCase())
      q"val $varName: Person = new Person($n)"
    }

    c.Expr[People](q"new People { ..$defs }")
  }
}

The macro finds all the introduced names by pattern matching against the expanded dynamic calls and generates the desired output code. Notice that the macro must be whitebox in order to be allowed to return an expression of a type derived from the one declared in the signature.

Answer 2

I don't think so. The expression you pass to a macro must be a valid Scala expression and identifiers should be defined.