Testing ClojureQL

After working on ClojureQL for some time, I've became tired of running manual tests all the time, even with the debug macro. So I began working on a test framework for this project. It's a combination of the test-is library and the concept of demos already built-in. For now we only have them, but they aren't very comprehensive and require a working connection for each backend you want to test. Furthermore, there's practically no validation of the results, we only know that the database accept the expression that have been sent. Now that we're approaching the release of version 1.0 we need something more flexible and exhaustive to keep the code stable.

The Plan

The idea of demo is not bad in itself, so I'll just complement it with tests that could be run without connection. It's all we need for day to day development on ClojureQL, the actual execution of the generated SQL code can be done only once in a while. To not break the DRY principle, I'll make them both use the same code, demos will be written using a special macro and tests will be generated automatically from them. The demos should also be improved by validating the results coming out of the tested databases.

The Prototype

Implementing this scheme directly into ClojureQL is not a simple task though, so I'll make a prototype to show how it works at a smaller scale. We'll use a very simple database written in Clojure and a ClojureQL-like DSL that compile to pseudo-SQL statements. No backends, no intermediate forms and only two kind of statements: inserts and selects. We'll walk through this prototype below, it's merely a hundred lines long.

The Database

All data is kept in a ref, *db*, which is a map of maps with strings as the sole data type for keys as well as for values. The only function we really need here is exec, which takes a compiled statement and returns a result if no exception is raised. I've also included a simple helper function, reset, to empty the database.

(def *db* (ref {}))

(defn reset [] (dosync (ref-set *db* {})))

(defn exec-insert [stmt]
  (let [tokens (drop 2 (seq (.split stmt " ")))
        table  (first tokens)
        values (apply hash-map (rest (rest tokens)))]
    (dosync (alter *db* assoc table
              (merge (@*db* table) values)))))

(defn exec-select [stmt]
  (let [tokens (drop 1 (seq (.split stmt " ")))
        table  (last tokens)
        keys   (take (- (count tokens) 2) tokens)]
    (map #((@*db* table) %) keys)))

(defn exec [stmt]
  (condp #(.startsWith %2 %1) stmt
    "INSERT INTO " (exec-insert stmt)
    "SELECT "      (exec-select stmt)
    (throw (SQLException. "error: unrecognized statement."))))

(defn insert [name & key-val-pairs]
  (apply str "INSERT INTO " name " VALUES " (interpose " " key-val-pairs)))

(defn select [name & keys]
  (str "SELECT " (apply str (interpose " " keys)) " FROM " name))

Our DSL is composed of the insert and select functions, which output the compiled pseudo-SQL code that the database can execute. It's a really dumb system, but we don't need more for the purpose of this experiment.

The Demos

The demos will remain the heart of the ClojureQL testing framework. The main objectives of the following code is to make it easy to define demos and to make their output human-readable. The trick is to also make them reusable, so demos will be defined as data, not code.

(defmacro defdemo [demo & stmts-results]
  `(def ~demo
     ~(vec (map #(vector (list 'quote (first %))
                         (list 'quote (second %)))
             (doall (partition 2 stmts-results))))))

(defdemo demo1
  (insert 'test1 'foo 1) {"test1" {"foo" "1"}}
  (select 'test1 'foo)   ("1"))

(defdemo demo2
  (insert 'test2 'foo 1 'bar 2) {"test2" {"bar" "2", "foo" "1"}}
  (select 'test2 'foo 'bar)     ("1" "2"))

(defn run-demo [name]
  (reset)
  (println "\n===" name "===\n")
  (doall
    (map (fn [[stmt expected-result]]
           (println "statement: " stmt)
           (let [compiled-stmt (eval stmt)
                 result        (exec compiled-stmt)]
             (println "compiled:  " compiled-stmt)
             (println "result:    " result "\n")
             (is (= result expected-result)))) (eval (symbol name)))))

(def demos ["demo1" "demo2"])

(defn run-demos []
  (every? identity
    (apply concat (map run-demo demos))))

The run-demo function test everything in a demo and provide a nice output. It returns a boolean that is the result of the assertion made at the end.

The Tests

We can now use the above to write some tests. The demos ensure us that the compiled statements are all valid. After that it's simply a matter of generating Clojure code, to define tests, and writing it to a file. The gen-test function will take care of the first part and write-tests the second one.

(defn gen-test [name]
  (let [demo (eval (symbol name))]
    `(deftest ~(symbol (.replace name "demo" "test"))
       ~@(map (fn [[stmt _]]
                `(is (= ~stmt ~(eval stmt)))) demo))))

(defn write-tests [filename]
  (when (run-demos)
    (with-open [writer (java.io.FileWriter. filename)]
      (binding [*out* writer]
        (newline)
        (doseq [demo demos]
          (pprint (gen-test demo))
          (newline))))))

(write-tests "tests.clj")

(load-file "tests.clj")

Once the tests file written, we can load it to run the tests it contains. So now we can use run-demos to test everything and run-tests to test compilation only.

Conclusion

There's still many details to figure out before implementing this framework. The namespace layout to be used for example or how to integrate the test-is library depending on what Clojure version we end up targeting. This is a work in progress so I'll keep this post updated until it's done. In the meantime, any comments or suggestions are welcomed.

Happy New Year!