Automatic spec test check generation does not work

[denistakeda]

Hello everyone and thank you for this awesome library!

I'm having trouble check my specs with kaocha. Basically it just does not work. tests.edn

#kaocha/v1
{:tests [{:kaocha.testable/type :kaocha.type/spec.test.check
          :kaocha.testable/id   :spec}
         {:kaocha.testable/type :kaocha.type/clojure.test
          :kaocha.testable/id   :unit}]}

project.clj

(defproject stest "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
            :url "https://www.eclipse.org/legal/epl-2.0/"}
  :dependencies [[org.clojure/clojure "1.10.0"]]
  :profiles {:kaocha {:dependencies [[lambdaisland/kaocha "0.0-554"]
                                     [org.clojure/test.check "0.9.0"]]}}
  :aliases {"kaocha" ["with-profile" "+kaocha" "run" "-m" "kaocha.runner"]}
  :repl-options {:init-ns stest.core})

The error output:

➜  stest git:(master) lein kaocha
[(EE)][(.)]
Randomized with --seed 992443957

ERROR in stest.core/ok-fn (form-init7457699483299345752.clj:1)
Uncaught exception, not in assertion.
Exception: java.util.concurrent.ExecutionException: Syntax error compiling at (clojure/test/check/clojure_test.cljc:95:1).
 at java.util.concurrent.FutureTask.report (FutureTask.java:122)
    java.util.concurrent.FutureTask.get (FutureTask.java:191)
    ...
    kaocha.type.spec.test.fdef$eval4786$fn__4789$fn__4793.invoke (fdef.clj:66)
    kaocha.type.spec.test.fdef$eval4786$fn__4789.invoke (fdef.clj:63)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2634$fn__2635.invoke (capture_output.clj:83)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2634.doInvoke (capture_output.clj:83)
    ...
    kaocha.testable$run.invokeStatic (testable.clj:119)
    kaocha.testable$run.invoke (testable.clj:110)
    ...
    kaocha.testable$run_testable.invokeStatic (testable.clj:189)
    kaocha.testable$run_testable.invoke (testable.clj:148)
    kaocha.testable$run_testables.invokeStatic (testable.clj:202)
    kaocha.testable$run_testables.invoke (testable.clj:192)
    kaocha.type.spec.test.ns$eval4850$fn__4851.invoke (ns.clj:45)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2634$fn__2635.invoke (capture_output.clj:83)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2634.doInvoke (capture_output.clj:83)
    ...
    kaocha.testable$run.invokeStatic (testable.clj:119)
    kaocha.testable$run.invoke (testable.clj:110)
    ...
    kaocha.testable$run_testable.invokeStatic (testable.clj:189)
    kaocha.testable$run_testable.invoke (testable.clj:148)
    kaocha.testable$run_testables.invokeStatic (testable.clj:202)
    kaocha.testable$run_testables.invoke (testable.clj:192)
    kaocha.test_suite$run.invokeStatic (test_suite.clj:7)
    kaocha.test_suite$run.invoke (test_suite.clj:5)
    kaocha.type.spec.test.check$eval4906$fn__4907.invoke (check.clj:48)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2634$fn__2635.invoke (capture_output.clj:83)
    ...
    kaocha.plugin.capture_output$capture_output_wrap_run_hook$fn__2634.doInvoke (capture_output.clj:83)
    ...
    kaocha.testable$run.invokeStatic (testable.clj:119)
    kaocha.testable$run.invoke (testable.clj:110)
    ...
    kaocha.testable$run_testable.invokeStatic (testable.clj:189)
    kaocha.testable$run_testable.invoke (testable.clj:148)
    kaocha.testable$run_testables.invokeStatic (testable.clj:202)
    kaocha.testable$run_testables.invoke (testable.clj:192)
    kaocha.api$run$fn__3129$fn__3130$fn__3131.invoke (api.clj:106)
    ...
    kaocha.api$run$fn__3129$fn__3130.invoke (api.clj:98)
    ...
    kaocha.api$run$fn__3129.invoke (api.clj:97)
    ...
    kaocha.api$run.invokeStatic (api.clj:83)
    kaocha.api$run.invoke (api.clj:71)
    kaocha.runner$run.invokeStatic (runner.clj:128)
    kaocha.runner$run.invoke (runner.clj:70)
    kaocha.runner$_main_STAR_.invokeStatic (runner.clj:161)
    kaocha.runner$_main_STAR_.doInvoke (runner.clj:140)
    ...
    kaocha.runner$_main.invokeStatic (runner.clj:172)
    kaocha.runner$_main.doInvoke (runner.clj:170)
    ...
    user$eval140.invokeStatic (form-init7457699483299345752.clj:1)
    user$eval140.invoke (form-init7457699483299345752.clj:1)
    ...
Caused by: clojure.lang.Compiler$CompilerException: Syntax error compiling at (clojure/test/check/clojure_test.cljc:95:1).
#:clojure.error{:phase :compile-syntax-check, :line 95, :column 1, :source "clojure/test/check/clojure_test.cljc"}
 at clojure.lang.Compiler.load (Compiler.java:7647)
    ...
    clojure.test.check$eval5886$loading__6706__auto____5887.invoke (check.cljc:10)
    clojure.test.check$eval5886.invokeStatic (check.cljc:10)
    clojure.test.check$eval5886.invoke (check.cljc:10)
    ...
    clojure.spec.gen.alpha$dynaload$fn__2628.invoke (alpha.clj:22)
    clojure.spec.gen.alpha$dynaload.invokeStatic (alpha.clj:21)
    clojure.spec.gen.alpha$dynaload.invoke (alpha.clj:17)
    clojure.spec.gen.alpha$fn__2632.invokeStatic (alpha.clj:29)
    clojure.spec.gen.alpha/fn (alpha.clj:29)
    ...

I created the minimal reproduction repo: https://github.com/denistakeda/stest

[plexus]

Hi @denistakeda, did you look at https://github.com/lambdaisland/kaocha/issues/121 ? There are still some problems with this plugin/test type. We will likely split it up into its own project and try to make it a bit more reliable. Any help with that is more than welcome! cc @WhittlesJr

[WhittlesJr]

Huh, weird... I use this regularly without issues, and it looks like your reproduction repo is just trying to do the same thing that is covered by our unit tests (which pass). I'll dig into this soon to see what the issue might be.

[WhittlesJr]

Nothing obvious after a cursory look. For my own reference when I come back to this: the reproduction repo is using lein, and I'm using tools-deps. It's also not using kaocha.plugin.alpha/spec-test-check, which I do use.

[emlyn]

I had this error too, and updating test.check to 0.10.0 fixed it.

[alysbrooks]

@denistakeda I re-ran your failing example (very helpful, btw), and updating test.check fixed it there, too. Can you confirm? Thanks!

[alysbrooks]

I think it's safe to close it after two years.