After working on ClojureQL for some time, I've became tired of running
manual tests all the time, even with the
debug macro.
So I began working on a test framework for this project. It's a
combination of the test-is library and the concept of demos already
built-in. For now we only have them, but they aren't very comprehensive
and require a working connection for each backend you want to
test. Furthermore, there's practically no validation of the results, we
only know that the database accept the expression that have been sent.
Now that we're approaching the release of version 1.0 we need something
more flexible and exhaustive to keep the code stable.
The Plan
The idea of demo is not bad in itself, so I'll just complement it with
tests that could be run without connection. It's all we need for day to
day development on ClojureQL, the actual execution of the generated SQL
code can be done only once in a while. To not break the DRY principle,
I'll make them both use the same code, demos will be written using a
special macro and tests will be generated automatically from them. The
demos should also be improved by validating the results coming out of
the tested databases.
The Prototype
Implementing this scheme directly into ClojureQL is not a simple task
though, so I'll make a prototype to show how it works at a smaller
scale. We'll use a very simple database written in Clojure and a
ClojureQL-like DSL that compile to pseudo-SQL statements. No backends,
no intermediate forms and only two kind of statements: inserts
and selects. We'll walk through this prototype below, it's merely a
hundred lines long.
The Database
All data is kept in a ref, *db*, which is a map of maps with
strings as the sole data type for keys as well as for values. The only
function we really need here is exec, which takes a compiled
statement and returns a result if no exception is raised. I've also
included a simple helper function, reset, to empty the database.
(def *db* (ref {}))
(defn reset [] (dosync (ref-set *db* {})))
(defn exec-insert [stmt]
(let [tokens (drop 2 (seq (.split stmt " ")))
table (first tokens)
values (apply hash-map (rest (rest tokens)))]
(dosync (alter *db* assoc table
(merge (@*db* table) values)))))
(defn exec-select [stmt]
(let [tokens (drop 1 (seq (.split stmt " ")))
table (last tokens)
keys (take (- (count tokens) 2) tokens)]
(map #((@*db* table) %) keys)))
(defn exec [stmt]
(condp #(.startsWith %2 %1) stmt
"INSERT INTO " (exec-insert stmt)
"SELECT " (exec-select stmt)
(throw (SQLException. "error: unrecognized statement."))))
(defn insert [name & key-val-pairs]
(apply str "INSERT INTO " name " VALUES " (interpose " " key-val-pairs)))
(defn select [name & keys]
(str "SELECT " (apply str (interpose " " keys)) " FROM " name))
Our DSL is composed of the insert and select functions,
which output the compiled pseudo-SQL code that the database can
execute. It's a really dumb system, but we don't need more for the
purpose of this experiment.
The Demos
The demos will remain the heart of the ClojureQL testing framework. The
main objectives of the following code is to make it easy to define demos
and to make their output human-readable. The trick is to also make them
reusable, so demos will be defined as data, not code.
(defmacro defdemo [demo & stmts-results]
`(def ~demo
~(vec (map #(vector (list 'quote (first %))
(list 'quote (second %)))
(doall (partition 2 stmts-results))))))
(defdemo demo1
(insert 'test1 'foo 1) {"test1" {"foo" "1"}}
(select 'test1 'foo) ("1"))
(defdemo demo2
(insert 'test2 'foo 1 'bar 2) {"test2" {"bar" "2", "foo" "1"}}
(select 'test2 'foo 'bar) ("1" "2"))
(defn run-demo [name]
(reset)
(println "\n===" name "===\n")
(doall
(map (fn [[stmt expected-result]]
(println "statement: " stmt)
(let [compiled-stmt (eval stmt)
result (exec compiled-stmt)]
(println "compiled: " compiled-stmt)
(println "result: " result "\n")
(is (= result expected-result)))) (eval (symbol name)))))
(def demos ["demo1" "demo2"])
(defn run-demos []
(every? identity
(apply concat (map run-demo demos))))
The run-demo function test everything in a demo and provide a nice
output. It returns a boolean that is the result of the assertion made at
the end.
The Tests
We can now use the above to write some tests. The demos ensure us that
the compiled statements are all valid. After that it's simply a matter
of generating Clojure code, to define tests, and writing it to a
file. The gen-test function will take care of the first part and
write-tests the second one.
(defn gen-test [name]
(let [demo (eval (symbol name))]
`(deftest ~(symbol (.replace name "demo" "test"))
~@(map (fn [[stmt _]]
`(is (= ~stmt ~(eval stmt)))) demo))))
(defn write-tests [filename]
(when (run-demos)
(with-open [writer (java.io.FileWriter. filename)]
(binding [*out* writer]
(newline)
(doseq [demo demos]
(pprint (gen-test demo))
(newline))))))
(write-tests "tests.clj")
(load-file "tests.clj")
Once the tests file written, we can load it to run the tests it
contains. So now we can use run-demos to test everything and
run-tests to test compilation only.
Conclusion
There's still many details to figure out before implementing this
framework. The namespace layout to be used for example or how to
integrate the test-is library depending on what Clojure version we end up
targeting. This is a work in progress so I'll keep this post updated
until it's done. In the meantime, any comments or suggestions are
welcomed.
Happy New Year!
