git-test
git-test copied to clipboard
Feature request: allow multiple `-t` arguments / add a `--all` argument
Currently it's only possible to run a single test at a time, at least without multiple invocations of git-test. It'd be useful to me to be able to specify multiple tests to run at the same time. I'd see this working like so:
-
git test run <commit>...
: Runs thedefault
test on the specified commits. -
git test run -t alpha -t beta <commit>...
: Runs the testsalpha
andbeta
on the specified commits. -
git test run -t default -t alpha <commit>...
: Runs the testsdefault
andalpha
on the specified commits. -
git test run --all <commit>...
: Runs all the defined tests, includingdefault
if defined, on the specified commits.
There's a small-but-greater-than-zero chance I'll manage a PR myself for this, but given the other projects currently on my plate I don't want to promise anything.
I wanted something similar and hacked in a solution the other day (https://github.com/veo-jdowner/git-test/tree/working). It is obviously not the right solution but it would be good to get some guidance @mhagger on how you would like it added if you support this feature.
In my case, I wanted syntax like,
git test run --test=foo,bar,baz ...
but the functionality is the important thing.
Thanks for your comments and the spike implementation! Sorry I haven't been more responsive but I've been very busy lately.
I think this is a great idea and it's one that I'd use myself. Let's talk a little bit about the requirements. When multiple tests are specified:
-
git test run
should run each of the tests, in the order specified, against each of the commits specified. A commit must pass every one of the tests for it to be considered to pass as a whole. As soon as a commit fails one test, that commit should be considered to have failed, and no more tests should be run against it (unless--keep-going
was specified). The results of each test should be recorded individually, test by test. If any of the tests has already been run against one of the commits' trees, then its old result should usually be reused. If one of the specified tests has already failed against the current commit, then normally that result should be printed and no tests should be run against that commit. If--retest
is specified, then that option should only apply to any tests that have stored failing results. For example, if I'm running testsa
,b
, andc
, and the current tree has a success result stored for testa
, a failing result stored for testb
, and no result stored for testc
, then testsb
andc
should be run but nota
. On the other hand,--force
and--forget
should apply to all tests. If--keep-going
is specified and a test fails against a certain commit, then the other tests should nevertheless be run against that commit before proceeding to subsequent commits.git test run
andgit test results
should display the results for each test separately. Especially, if there is a failure,git test run
should make it obvious which test failed. It might be nice forgit test results
to list the results in a table with one line per commit rather than printing one line per(tree, test)
pair. -
git test add
should error out if more than one test is specified. -
git test forget-results
should forget the results for all tests that are specified. (We could live without this feature.) -
git test remove
should remove all of the tests that are specified. (We could live without this feature.)
I don't have a preference between -t test1 -t test2
vs. -t test1,test2
. In fact, it wouldn't be hard to support both syntaxes and even -t test1 -t test2,test3
by having the option append arg.split(',')
to a list.
As for how to implement it, my first inclination is to make Test
into an abstract base class, with one implementation being a NamedTest
that has the current behavior. But there would also be a CompoundTest
implementation, which contains an array of Test
describing the subtests. There would be a new function like make_test()
that returns a single NamedTest
if only one case was specified, or a CompoundTest
if more than one. The CompoundTest
class would implement many of its methods by looping over its array of subtests and calling the corresponding method on each one, combining the results as appropriate and returning them to the original caller. In this way it might be possible to add support for multiple tests without having to change much of the existing code.
I haven't checked carefully whether this will really work. Can all of the Test
methods really be implemented by CompoundTest
and give the desired behavior? It might be necessary to add a couple more methods to the Test
interface to get the right semantics.
It would be great if somebody wants to work on this. Realistically, I don't expect to have time for it anytime soon.