Configuring opts for kaocha.type/spec.test.check

[baumandm]

I'm having trouble successfully passing opts to spec.test.check as described in the documentation: https://cljdoc.org/d/lambdaisland/kaocha/0.0-554/doc/automatic-spec-test-check-generation. I've tried various combinations with namespaces, without, with checks or without, and it doesn't seem to take effect. Here's an example configuration:

#kaocha/v1
{:tests [{:type :kaocha.type/spec.test.check
          :id   :generative-fdef-checks
          :checks [{:opts {:num-tests 5}}]}]

 :plugins [:kaocha.plugin.alpha/spec-test-check]
 }

Just using lein kaocha, the opts do not take effect. However, I am able to override this successfully via the CLI since I have the plugin added as well:

lein kaocha --stc-num-tests 5

I assume I'm doing something wrong in my config but I have no idea what. Any ideas?

[plexus]

cc @WhittlesJr

[WhittlesJr]

@baumandm, have you tried removing the plugin? The plugin only serves to set the options to default values. I'm pretty sure it's mutually exclusive with specifying test suites yourself.

[WhittlesJr]

Hmm, well I see how the documentation says you should be able to use both... let me check something.

[WhittlesJr]

OK yeah I'm pretty sure the plugin overrides ::clojure.spec.test.check/opts with the CLI values, even if there are no values provided. This isn't ideal, so I'm going to look into a PR to make this work as you'd expect.

[baumandm]

I did try without the plugin added, and was still not able to get opts to work via tests.edn.

[WhittlesJr]

Ah, alright. I'll add some tests and try to figure out a fix. I don't personally use tests.edn so I never noticed the issue.

[WhittlesJr]

Looks like it works if you put :clojure.spec.test.check/opts at the top-level, but not otherwise. PR is forthcoming.

[baumandm]

That's one thing I didn't try earlier, but I can confirm this works:

#kaocha/v1
{:tests [{:type :kaocha.type/spec.test.check
          :id   :generative-fdef-checks}]
 :clojure.spec.test.check/opts { :num-tests 5 }}

The docs seem to indicate one of these two options instead, both of which don't work currently:

#kaocha/v1
{:tests [{:type :kaocha.type/spec.test.check
          :id   :generative-fdef-checks
          :checks [{:clojure.spec.test.check/opts { :num-tests 5 }}]}]}

or

#kaocha/v1
{:tests [{:type :kaocha.type/spec.test.check
          :id   :generative-fdef-checks
          :clojure.spec.test.check/opts { :num-tests 5 }}]}

Thanks for your help on this!

[WhittlesJr]

Ah, one problem is that :checks is not auto-resolving to :kaocha.spec.test.check/checks.

Also, opinion question: does the aforementioned config option seem awkward to you? (and @plexus?) In other words, would it make more sense to simply specify different :tests, instead of this nested :checks value? I can't remember why I decided to add :checks.

[WhittlesJr]

Wouldn't it make more sense to use the existing filtering / skipping syntax at the test suite level? I'd have to implement them for function vars rather than test vars, but it would seem to me to be more consistent that way with the rest of the library.

[WhittlesJr]

So this would be the syntax I'd propose:

{:tests [{:type     :kaocha.type/spec.test.check
          :id       :fn-1-generative-tests
          :focus    ['sample/fn-1]
          :stc-opts {:num-tests 1}}]}

[baumandm]

This looks good to me. What would you do with the other per-check options like instrument? and check-asserts?. Move them to the same location?

One question is, how would one run generative tests for all namespaces, but override the opts for a specific namespace/function? Would something like this detect and avoid duplicates?

{:tests [{:type     :kaocha.type/spec.test.check
          :id       :fn-1-generative-tests
          :focus    ['sample/fn-1]
          :stc-opts {:num-tests 1}}
         {:type     :kaocha.type/spec.test.check
          :id       :remaining-generative-tests}]}

[WhittlesJr]

I believe you could :focus the special functions in one test case and :skip them in the other. But you'd be maintaining the list in two places. Alternatively, you could use metadata tags instead of a list of functions, like so:

{:tests [{:type       :kaocha.type/spec.test.check
          :id         :slow-generative-tests
          :focus-meta [:slow-gen]
          :stc-opts   {:num-tests 10}}
         {:type      :kaocha.type/spec.test.check
          :id        :generative-tests
          :skip-meta [:slow-gen]}]}

And just decorate your functions with ^:slow-gen when needed.

@plexus, would you approve such a change?

[plexus]

I have to say I still haven't really used this functionality and am only superficially familiar with the implementation...

Wouldn't it make more sense to use the existing filtering / skipping syntax at the test suite level?

That seems like a good idea, leverage people's existing mental model.

          :focus    ['sample/fn-1]

No need to quote here, tests.edn is read, not evaluated.

I believe you could :focus the special functions in one test case and :skip them in the other.

Correct, you can combine :focus/:skip like this, or :focus-meta/:skip-meta. I've done that before on projects. The only caveat is that when you then add --focus/--skip/--focus-meta/--skip-meta on the command line these will get added together and form a logical or which sometimes leads to unexpected result. e.g. if you want to run just a single test with --focus testable-id this will instead run that test plus all tests with :focus-meta [:slow-gen].

@WhittlesJr how would you feel about splitting this plugin and test type into its own project? it can still live under github.com/lambdaisland/kaocha-*, but I can give you push and release rights. That way the project can evolve at its own pace, separate from kaocha releases, and it would give it a higher degree of discoverability.

[WhittlesJr]

I'd be up for splitting the project

[alysbrooks]

I think we'd be still be open to a PR splitting these, but wouldn't be a project we'd tackle in the short term. Let us know @WhittlesJr if that's still something you'd consider. (No hurry.)