kaocha
kaocha copied to clipboard
lambdaisland
Exception when running tests with fail-fast and watch
After a test failure, I edit the test, and then I get the following output:
lambdaisland/kaocha {:mvn/version "0.0-529"}
[watch] Reloading #{orgmode.parser-test}
[watch] Re-running failed tests #{:orgmode.parser-test/parser-tests}
ERROR[watch] Fatal error in test run #error {
:cause throw+: #:kaocha{:fail-fast true}
:data #:kaocha{:fail-fast true}
:via
[{:type clojure.lang.ExceptionInfo
:message throw+: #:kaocha{:fail-fast true}
:data #:kaocha{:fail-fast true}
:at [slingshot.support$stack_trace invoke support.clj 201]}]
:trace
[[slingshot.support$stack_trace invoke support.clj 201]
[kaocha.report$fail_fast invokeStatic report.clj 338]
[kaocha.report$fail_fast invoke report.clj 331]
[kaocha.config$resolve_reporter$fn__2992$fn__2993 invoke config.clj 142]
[clojure.core$run_BANG_$fn__8790 invoke core.clj 7715]
[clojure.lang.ArrayChunk reduce ArrayChunk.java 63]
[clojure.core.protocols$fn__8154 invokeStatic protocols.clj 136]
[clojure.core.protocols$fn__8154 invoke protocols.clj 124]
[clojure.core.protocols$fn__8114$G__8109__8123 invoke protocols.clj 19]
[clojure.core.protocols$seq_reduce invokeStatic protocols.clj 31]
[clojure.core.protocols$fn__8146 invokeStatic protocols.clj 75]
[clojure.core.protocols$fn__8146 invoke protocols.clj 75]
[clojure.core.protocols$fn__8088$G__8083__8101 invoke protocols.clj 13]
[clojure.core$reduce invokeStatic core.clj 6828]
[clojure.core$run_BANG_ invokeStatic core.clj 7710]
[clojure.core$run_BANG_ invoke core.clj 7710]
[kaocha.config$resolve_reporter$fn__2992 invoke config.clj 142]
[kaocha.api$resolve_reporter$fn__3084 invoke api.clj 56]
[kaocha.monkey_patch$report invokeStatic monkey_patch.clj 27]
[kaocha.monkey_patch$report invoke monkey_patch.clj 26]
[kaocha.monkey_patch$do_report invokeStatic monkey_patch.clj 70]
[kaocha.monkey_patch$do_report invoke monkey_patch.clj 35]
[kaocha.watch$plugin_pre_test_hook invokeStatic watch.clj 173]
[kaocha.watch$plugin_pre_test_hook invoke watch.clj 150]
[clojure.lang.AFn applyToHelper AFn.java 156]
[clojure.lang.AFn applyTo AFn.java 144]
[clojure.core$apply invokeStatic core.clj 667]
[clojure.core$apply invoke core.clj 660]
[kaocha.plugin$run_hook_STAR_$fn__1230 invoke plugin.clj 42]
[clojure.lang.PersistentVector reduce PersistentVector.java 343]
[clojure.core$reduce invokeStatic core.clj 6827]
[clojure.core$reduce invoke core.clj 6810]
[kaocha.plugin$run_hook_STAR_ invokeStatic plugin.clj 40]
[kaocha.plugin$run_hook_STAR_ doInvoke plugin.clj 39]
[clojure.lang.RestFn applyTo RestFn.java 146]
[clojure.core$apply invokeStatic core.clj 671]
[clojure.core$apply invoke core.clj 660]
[kaocha.plugin$run_hook invokeStatic plugin.clj 51]
[kaocha.plugin$run_hook doInvoke plugin.clj 50]
[clojure.lang.RestFn invoke RestFn.java 442]
[kaocha.testable$run_testable invokeStatic testable.clj 149]
[kaocha.testable$run_testable invoke testable.clj 148]
...
Please provide as much context as possible when reporting issues
- what's your tests.edn?
- what command line flags are you using?
- are you running from CLI or REPL?
- what does the failing code look like?
Thanks
- tests.edn:
{:tests [{:id :all
:test-paths ["src" "test"]}]
:watch? true
:plugins [:kaocha.plugin/notifier]
:reporter kaocha.report/documentation}
- Command line flags:
bin/kaocha --reporter kaocha.report/documentation --fail-fast --no-randomize
-
Running in CLI.
-
Not sure what is failing - cannot figure out what the exception means, whether it is an error in my code or an error in kaocha. i.e what does the following mean?
Fatal error in test run #error {
:cause throw+: #:kaocha{:fail-fast true}
:data #:kaocha{:fail-fast true}
This has to do with the fail-fast mechanism, kaocha generates an exception to halt execution, rather than continuing the current test var. This is normally handled by the leaf test types, but it seems in this case the exception is being raised outside of that scope. Looking at the line numbers it seems to be in a pre-test hook.
(t/do-report {:type :error
:kaocha/testable {:kaocha.testable/id :kaocha/watch}
:message (str "Failed reloading ns: " (::error-ns test-plan))
:actual error})
So that's probably the thing that triggered this. You have a namespace that isn't reloading cleanly.
We should fix this by having *fail-fast?* be false outside of the run phase. The fail-fast reporter should also add the original error to the exception data so it's more clear where issues like these originate.
Thanks for the info. The issue that caused the problem seems to be fixed now on my side - not sure what caused it, but getting all my tests to pass sorted it out.
However, it might be worth keeping this issue open as the current error is unfriendly and probably worth fixing.
Also, unrelated, but thanks for Kaocha, it works really well and looks like a well thought out system. Just started using it today.
This definitely needs fixing, it might not be super common as it's really a combination of watch+fail-fast+load errors, but it will affects plugins/hooks more generally.
I'm pretty busy right now with other stuff, but I'll probably make more time for Kaocha again coming fall.
I hit something like this using greenlight-kaocha. it only failed in CI (could not reproduce locally) and fail-fast masked the failing assertion so it was hard to find the culprit. when I saw this bug I tried disabling fail-fast and then I could see the problem.
I'm inclined to leave this issue open because it was never really addressed (although it's possible we fixed it inadvertently), even though it appears to be a very rare bug.
