Running tests at the REPL

[mfikes]

With the Cognitect test runner, you can run all your tests while via the REPL by simply doing

(require '[cognitect.test-runner])
(cognitect.test-runner/test {})

I was wondering if the same could be done with cljs-test-runner.

An experiment shows that it is probably possible via a macro.

https://github.com/mfikes/cljs-test-runner/tree/repl

The output of this experiment is below.

Before delving into this too far, wanted to share and see what your thoughts are.

Mikes-iMac:cljs-bean mfikes$ clj -A:repl/node
ClojureScript 1.10.520
cljs.user=> (require 'cljs-bean.core-test)
nil
cljs.user=> (require 'cljs-test-runner.test-runner)
nil
cljs.user=> (cljs-test-runner.test-runner/run-tests {})
WARNING: Use of undeclared Var doo.runner/set-entry-point! at line 31 cljs-test-runner-out/gen/cljs_test_runner/gen.cljs
WARNING: Use of undeclared Var doo.runner/karma? at line 31 cljs-test-runner-out/gen/cljs_test_runner/gen.cljs
WARNING: Use of undeclared Var jx.reporter.karma/start at line 31 cljs-test-runner-out/gen/cljs_test_runner/gen.cljs

Testing cljs-bean.core-test
{:result true, :num-tests 100, :seed 1561756175726, :time-elapsed-ms 262, :test-var "roundtrip-1"}
{:result true, :num-tests 100, :seed 1561756175989, :time-elapsed-ms 144, :test-var "roundtrip-2"}
{:result true, :num-tests 100, :seed 1561756176133, :time-elapsed-ms 102, :test-var "roundtrip-3"}
{:result true, :num-tests 100, :seed 1561756176235, :time-elapsed-ms 91, :test-var "roundtrip-4"}

Ran 72 tests containing 510 assertions.
0 failures, 0 errors.
0

[Olical]

Oh that's really cool! I'm totally okay with this, it's a small change and it looks like it'll be a great addition for some people. Namely you :smile:

If you'd like to submit a PR for this I'll give it a once over, I can't see why this can't be part of the main project though, it's not really bloat. Great work, as always!

Edit: And thank you for checking!

[mfikes]

@Olical Thanks. Since you are cool with this, I'll put together a proper PR for review.

[Olical]

Hey Mike, just wondering if you ever had this code working? I'd love to see what you were thinking about even if it doesn't work? I can maybe pick this torch up eventually.

[mfikes]

@Olical Indeed there was some issue that I encountered when trying to solve this that was difficult to overcome. I failed to record it but I'll see if I can find the original branch and re-construct what the problem was.

[mfikes]

@Olical OK, I took a look to see where I had become stuck. The experimental branch where I was progressing at the time is https://github.com/mfikes/cljs-test-runner/tree/repl

One salient aspect of the design being pursued there is that, since you are already in an established REPL, there is no need to involve doo to set up a ClojureScript / JavaScript runtime within which to run tests: In the REPL case, all that is needed is to (a) dynamically generate the synthetic namespace as cljs-test-runner already does, and (b) to simply require the synthetically generated namespace.

Another salient aspect is that I was ignoring for the time being any concerns surrounding self-hosted ClojureScript, and instead just focusing on trying to get conventional JVM ClojureScript working.

The macro in that branch is set up to do parts (a) and (b) as described above. The part where I was getting stuck was (b): The macro essentially expands into a require form:

(cljs-test-runner.test-runner/run-tests {})

would, essentially expand to be

(require 'cljs-test-runner.gen :reload)

In fact, if you were to issue that particular require in a REPL, the tests would run. But they don't and instead you get an error. This error is easy to reproduce independently of this project:

If you were to evaluate (require 'clojure.set) but instead first look at its macro expansion and instead directly evaluate its expansion: (ns* (:require 'clojure.set)) you will see an error. Digging a little, the reason is that if you essentially evaluate an ns* form in the REPL, the Clojure implementation of the REPL first wraps the entire form in a construct that will set *1, handle exceptions, etc., and then evaluate that wrapped form. The problem with this is that the resulting ns* form is no longer a top-level form, but instead is buried inside the wrapping form. And since it is not a top-level form, something down in the ClojureScript analyzer or compiler is not happy.

You can easily cause the same error directly in a REPL if you, for example supply your own wrapping: (do (require 'clojure.set))

This is essentially what derails the attempt to evaluate (cljs-test-runner.test-runner/run-tests {}).

Now, given this, the REPL is hard-coded to know not to wrap a form for which the first symbol is require, amongst other symbols. If you revise the REPL code to do the same for the symbol cljs-test-runner.test-runner/run-tests, you can get things to work.

Here is an example:

cljs.user=> (require 'cljs-test-runner.test-runner)
nil
cljs.user=> (cljs-test-runner.test-runner/run-tests {})

Testing cljs-bean.core-test
{:result true, :num-tests 100, :seed 1571595381223, :time-elapsed-ms 298, :test-var "roundtrip-1"}
{:result true, :num-tests 100, :seed 1571595381522, :time-elapsed-ms 145, :test-var "roundtrip-2"}
{:result true, :num-tests 100, :seed 1571595381667, :time-elapsed-ms 133, :test-var "roundtrip-3"}
{:result true, :num-tests 100, :seed 1571595381801, :time-elapsed-ms 163, :test-var "roundtrip-4"}
{:result true, :num-tests 100, :seed 1571595381964, :time-elapsed-ms 170, :test-var "roundtrip-5"}

Testing cljs-bean.from.cljs.core-test

Testing cljs-bean.transit-test

Ran 111 tests containing 735 assertions.
0 failures, 0 errors.
nil
cljs.user=>

So, now knowing the root of the problem, the challenge is to devise some sort of solution for it. Perhaps instead of having the macro evaluate to a require form, it could make calls upon the ClojureScript compiler to attempt to cause the same side effect, or maybe there is some clever way to set the :wrap option for REPLs so that cljs-test-runner.test-runner/run-tests or run-tests is detected to be this particular macro and to not wrap it.

So that's where things sit now.

If you wanted to try hacking on it some to see if progress can be made, let me know.

If you decide to go down that path, feel free to take my branch with its experimental changes as a basis for hacking---you can see evidence of the things I was trying and some commented out code and bits that are hard coded in places.

Otherwise, I may get back to this at some point.

(I do feel that this feature would be a really nice thing to have in general---you often want to run all of your tests while at the REPL.)