Passing in Dynamic Number of "Variables" to Datomic API Functions

Question

This is a repost of a question I asked on the Datomic google group (see here - last 3 posts are best).

This problem revolves around the fact that I'm dynamically generating Datomic queries. So the number of parameters (names and values) is not known until passed into a function. I can generate the query just fine. The core problem is that, using the Clojure Datomic API, I cannot call the datomic.api/q function with a dynamic number of arguments. So the below ~@unquote-splice expression fails. I tried several other approaches, including partial and apply, to no avail.

(def expression-final `(datomic.api/q ~expression-intermediate ~db-conn ~@param-values))   ;; gives the error in question
(eval expression-final)


 java.lang.Exception: processing rule: (q__34868 ?e)
    at datomic.datalog$eval_rule$fn__4687.invoke(datalog.clj:977)
    at datomic.datalog$eval_rule.invoke(datalog.clj:957)
    at datomic.datalog$eval_query.invoke(datalog.clj:999)
    at datomic.datalog$qsqr.invoke(datalog.clj:1053)
    at datomic.datalog$qsqr.invoke(datalog.clj:1021)
    at datomic.query$q.invoke(query.clj:453)
    at datomic.api$q.doInvoke(api.clj:31)
    ... 1 stack levels elided ...
    at user$eval34866.invoke(crud_spec.clj:32)
    ... 3 stack levels elided ...
    at stefon_datomic.crud$retrieve_entity.invoke(crud.clj:95)
    ...
 Caused by: java.lang.Exception: processing clause: [?e :posts/title (quote ?title)]      ;; this fails with or without the (quote ...)
    at datomic.datalog$eval_clause$fn__4667.invoke(datalog.clj:934)
    at datomic.datalog$eval_clause.invoke(datalog.clj:900)
    at datomic.datalog$eval_rule$fn__4687.invoke(datalog.clj:972)
    at datomic.datalog$eval_rule.invoke(datalog.clj:957)
    ...
 Caused by: java.lang.UnsupportedOperationException: nth not supported on this type: Symbol
    ... 2 stack levels elided ...
    at datomic.datalog$extrel_coll$fn__4384.invoke(datalog.clj:197)
    ... 4 stack levels elided ...
    at datomic.datalog$iterator.invoke(datalog.clj:30)

I'm wondering if this is a bug in the Datomic Clojure API? Or if there's an easier way to pass in a dynamic number of variables. Hard coding the amount of variables passed in, defeats the purpose of dynamically generating the query. See the last 3 posts here, to get more detail.

Thanks

Answer 1

So, based on your source-code, I have managed to simplify the code a bit:

(defn add-entity-ns
  [ekey datom-map]
  (reduce-kv (fn [a k v]
               (assoc a (keyword
                         (name ekey)
                         (name k))
                      v))
             {}
             datom-map))

(defn retrieve-entity
  [conn constraint-map]
  (let [name-fn (comp symbol
                      (partial str "?")
                      name)
        param-names (map name-fn
                         (keys constraint-map))
        param-vals (vals constraint-map)
        constraint-map (add-entity-ns :posts constraint-map)
        where-clause (map #(vector '?e % %2)
                      (keys constraint-map)
                      param-names)
        in-clause (conj param-names '$)
        final-clause (concat [:find '?e]
                             [:in] in-clause
                             [:where] where-clause)]
    (apply d/q final-clause (d/db conn) param-vals)))

A note: When I started using datomic, I wrote myself similar functions to generate queries. I ended up throwing them away, for several reasons:

1. It's a mess. The Datalog is designed for doing logic queries. It takes significantly longer to generate a logic query and invoke it, even though datomic does caching. This is because Datalog has to resolve the query logically, which takes quite some time.
2. If you know how to query, it performs much better to actually query once for all entities and then filter them out, doing usual sequence transformation (and for a huge database, use reducers!). This performs on avg. 20x faster than Datalog (benchmarked using criterium). If you don't believe me, write a query that queries all your :post/x entities from the database, than filter out what posts match your criteria using filter and so on. If you have a large database, use reducers with foldcat.
3. I wrote myself a filtering engine (for bunches of entities) that serves as sort of a query-DSL. I modify it depending on the project and database structure. Being able to write DSLs is the power of LISP and Datomic makes it really possible thanks to entities being hash-map like data-structures. It performs way faster than Datalog (at the cost of being domain-specific).

Answer 2

Ok, I got this solved over on the Datomic Google Groups (see here). Basically, fig.1 works. And also, I'm able to run the generated code (fig.2) without the dreaded quoting. Hope this helps others.

(datomic.api/q '{:find [?e]
                 :in [$ [?title ?content-type]]
                 :where [[?e :posts/title ?title] [?e :posts/content-type ?content-type]]}
               (datomic.api/db conn)
               ["t" "c/t"])

fig.1

(datomic.api/q expression-intermediate-0 (datomic.api/db conn) (into [] param-values))

fig.2