kaocha
kaocha copied to clipboard
lambdaisland
Support for test-ns-hook
clojure.test has this feature where you can define a function named test-ns-hook, and instead of running the test vars in the current namespace, it will run that function.
My impression is that this is mostly used if you want to run your test vars in a very specific order, which in turn is something that's not often needed (and generally a smell).
Kaocha currently does not support this. The problem with it is that it makes things opaque, we lose all the rich information in the test plan regarding test vars, so e.g. filtering will not work.
The easiest way to make this work is to treat the test-ns-hook var as a single test var, and ignore any other test vars in that namespace. This should work, but it may produce sub-optimal results. e.g. say you have this code
(deftest aaa)
(deftest bbb)
(defn test-ns-hook []
(aaa)
(bbb))
These calls to (aaa) and (bbb) go through clojure.test/test-var, rather than through Kaocha's testable machinery, meaning events will not have the right :kaocha/testable associated with them. (this in a way makes sense, because there will be no testable for aaa or bbb, since they are ignored in favor of test-ns-hook).
There are probably (hacky) ways around this, but given how little this seems to be used I intend to start with the simple approach, so at least we have some (albeit suboptimal) support for this.
I am interested in this. My usescase actually not is to call them in a specific order.
Instead I have a set of tests that will have to be run twice with the same in and outputs. The only thing that changes is a setup function that is called in the :each fixture. So my idea is to call test-ns-hook, execute the test in the namespace one with setup function x and then a second time with setup function y.
It is very unlikely that Kaocha will ever support this. Why not turn the test into a function, and then have a deftest for each combination of setup function+test function?
Could this also be achieved through a hook that receives the deftest symbols in the namespace and returns them in a different order (i.e. in the order the test author needs because of dependencies between tests)? That would feel a little bit like Pedestal Interceptors can see and manipulate the chain of interceptors that will be executed to handle an HTTP request.
@devurandom Interesting idea. That would certainly fit our "data pipeline" architecture. It would also be quite flexible. For example, you could even do things like enforce a rough ordering. Or you could always make sure test A always runs sometime before test B but not care about the order otherwise. Or you could always run certain important tests first.
I'm not sure this will be a priority, though, because randomizing tests remains our recommendation, and it doesn't sound like there's a major use case for either test-ns-hook or the hook you propose. We would, as always, welcome a PR!
We use the hook for our end-to-end tests, where tests of certain parts of the system expect the system to be in a certain state from the tests that should have run before. Since these are end-to-end tests, we do not inject artificial data, and since each action taken by the previous tests takes quite a while it is not feasible to repeat the actions to create the state over and over again for each test.
Thank you for elaborating! I shouldn't have said "not a major use case." I think these use cases are infrequent, but if you need them, it's often unavoidable.
This ticket is about supporting test-ns-hook as it exists in clojure.test, which we consider legacy and are unlikely to ever support.
One can imagine multiple alternatives that would serve the same use case, and that would fit better with kaocha's architecture. These could well be done with third party plugins.
If someone has interest in developing such a plug-in we'd be happy to provide guidance.
Docs: For a description of test-ns-hook cf. sections "RUNNING TESTS" and "FIXTURES" of https://clojure.github.io/clojure/clojure.test-api.html.
I authored test-ns-hook Kaocha plugin. I think it is not perfect but better than nothing.
You can use it tentatively, until the Kaocha team implements this feature.
https://github.com/philoskim/kaocha-test-ns-hook
This is great, @philoskim. You're welcome to make a PR to link to your plugin from the Kaocha README.
If it's possible to do this with a plugin then I prefer that solution over having in Kaocha core.
To fix the issues you mention in your README with regards to the reporter output being incorrect, I think you can fix that by introducing a new test group type that sits in between namespace and var.
@plexus I will investigate your advice and implement it, if it is possible.