Not registering failures on async tests

[mauricioszabo]

I'm using Shadow-CLJS, and devcards, to test async code. If I try to use this library to test it, and there's a failure, it's not being registered. It's kinda tedious to reproduce this error, so I created a repository to reproduce it: https://github.com/mauricioszabo/bug-info-example

To simplify, a test with devcards like this one:

(defn some-code [a]
  (js/Promise. (fn [resolve]
                 (js/setTimeout #(resolve (+ a 10))
                                1000))))

(set! cards/test-timeout 8000)
(cards/deftest some-async-test
  (async done
    (.. (some-code 12)
        (then (fn [res] (is (match? 21 res))))
        (then done)
        (catch done))))

(cards/deftest correct-async-test
  (async done
    (.. (some-code 12)
        (then (fn [res] (is (= 21 res))))
        (then done)
        (catch done))))

Will produce the following HTML output:

On the test with matcher-combinators, neither the failure nor the assertion is registered.

[kommen]

I'm running into the same problem: mismatches using match? are not registered and when using async in cljs tests.

Looking at the report handlers, it looks like it is handled the same way as the :fail handlers in cljs.test, so it's not obvious to me what is wrong here:

https://github.com/nubank/matcher-combinators/blob/638b7c8dd0ceb881f4edcb38676c3b60ad0f6a10/src/cljc/matcher_combinators/cljs_test.cljc#L103-L121

[plexus]

The async is a red herring here, matcher-combinators does not work with devcards or kaocha-cljs in general:

(cards/deftest non-async-test
  (is (match? 22 12)))

Because neither of these test runners have implementations for the :matcher-combinators/mismatch message. The situation is similar for Clojure, but there Kaocha explicitly contains support for matcher combinators, I have written about this a little over here.

This is a flaw in the design of clojure.test/cljs.test. Tests generate events (through clojure.test/do-report), which are handled by a reporter. Custom assertion libraries can generate custom events, custom reporters provide custom rendering, but if you do both at the same time then you end with events that aren't known or understood by your reporter.

The implementation is a bit different in ClojureScript because there are no dynamic vars, so instead of rebinding clojure.test/report you use cljs.test/update-current-env! to change the reporter, which will then be used as part of the multimethod dispatch.

There isn't much to be done about this, besides patching devcards and kaocha-cljs (I will do the latter). What MC could try to do is detect if the muitimethod has an implementation for :matcher-combinators/mismatch, and if not fall back to a plain :fail message.

[philomates]

Hmm, I thought I merged @plexus's fix for the :matcher-combinators/mismatch issue (https://github.com/nubank/matcher-combinators/pull/49). Though looking at it I may have forgotten to update the cljs code :(

My cljs dev env seems to be busted and I'm a little busy at work but when I get a chance I'll try to dive into this issue. In the meantime, contributions are always welcome, especially for cljs, which I've never coded with other than in this lib

[plexus]

I see, it seems the CLJS version still uses ::mismatch: https://github.com/nubank/matcher-combinators/blob/638b7c8dd0ceb881f4edcb38676c3b60ad0f6a10/src/cljc/matcher_combinators/cljs_test.cljc#L47

[kommen]

I can confirm that @plexus' fix https://github.com/lambdaisland/kaocha-cljs/pull/23 in kaocha-cljs fixed the issue for me. Thanks!

[philomates]

https://github.com/nubank/matcher-combinators/pull/189 has been released in 3.8.2 and I think should address this. If not, let me know and I can look into it further