Feature: Run test suite on startup with `--watch` argument

[stroiman]

Hey, this is an amazing tool, and my primary use case is to have tests run automatically when saving a file. I tried gow, but that seems flaky, doesn't always run when files change.

This seems to be rock solid in that regards, and it's now my new default test runner front-end.

But when starting with --watch mode, tests don't run until I save a file, which is not a big issue, but I wouldn't call it a sensible default, and is also different from any other "watch" tool I can think of. They generally run the task on startup, and then again on file system events.

I'd love to see tests running on startup in watch mode, either as a default, as through a CLI argument.

[dnephin]

Glad to hear that gotestsum is working well for you! I use --watch almost daily as well!

The only challenge I see with running tests on startup with --watch is that --watch only runs tests for the one package that changed. This behaviour is essential on large projects. Some of the projects I used to work on would take 20+ minutes to run all the tests sequentially.

I think making it an option to run all tests on startup would be fine. I'd probably want to enable that with something like --watch=run-all-on-start, but a separate flag could also be ok.

Also, in case you missed it in the docs, pressing a in --watch mode will run all the tests once. That might also be an option.

[stroiman]

That's a good point - and that part isn't a big issue for me either. But doesn't go test under the hood cache the result, so it avoids rerunning unmodified tests (but then setting -count=1 disables the cache).

There was actually another point I wanted to make, and that is detecting changes in dependencies.

I would like to rerun tests when dependencies changed.

So far, my solution was to have a few different make targets for when working with different areas of the library, with a positive list of packages in the package options - but it's not exactly what I want, as it still runs all test suites in the monitored package; but I am only interested in the test suite of one package, but just rerun when a dep changes.

I used to use ginkgo which does actually have that behaviour when its own test runner is used. But I'm phasing out ginkgo, partly because it's a habit I picked up before Go had subtests, so the benefit somewhat diminished - but primarily because it makes it more difficult to attract collaborators.

My code base has a quite special case, as I support two JavaScript engines, V8 and Goja (native Go JS) - but the behaviour should be the same, so they share a common set of tests, but they are all mounted as subtests in a root test of the two main package. A change to the shared test library should rerun the suite.

But I didn't want to break the ice with that :)

---

So it was finding a replacement for ginkgo that implemented a watch mode. As mentioned, I first used gow, but as I mentioned, it doesn't always react to file save, but also has verbose output, as it prints a line for every package, including those without test suites. So that's what brought me to gotestwatch - which has nicer output, and a stable file system watcher.

[stroiman]

At any rate, a stable --watch is the most important feature. TDD for me is about feedback, it's making a change to the code, and verify that it has the expected outcome, and test runners picking up on file save is the most effective way of feedback for the vast majority of the code.

[dnephin]

A mode or keybinding to run "all impacted packages" would be awesome! I have wanted that feature as well, but never found the time to build it.

I think using go list it should be possible to find all the impacted packages, and run the tests for all of those.

[stroiman]

I wasn't aware of go list, but that made me solve half the problem in the shell, to generate the --packages argument automatically.

It's not exactly what I want, as it also runs the tests in dependencies - when I only want to focus on the tests in a single package. But it does save me the trouble of manually creating the --packages arg.

Here's the solution. The script processes the dependencies so only submodule dependencies are kept, and the module name replaced with a ..

#!/bin/sh

PREFIX="github.com/gost-dom/browser"

while IFS= read -r line; do
  if [[ "$line" == "$PREFIX"* ]]; then
    echo "${line/#$PREFIX/.} \c"
  fi
done

It's saved in the project root with execute permissions, is being executed by ./deps in the make target:

test-dom: 
	gotestsum --packages "`go list -deps ./dom | ./deps`" --format dots --watch

[stroiman]

Ah, and the PREFIX could be detected by go list without arguments, removing the need for a hardcoded value.

[stroiman]

Hey, :)

Your mentioning of go deps made me make a small shell script where you can specify a package to test (so far only one - unless multiple work magically out of the box), it will monitor all dependencies (inside the current module) for changes.

It uses nodemon, as that was the tool I knew to exist.

#!/bin/sh

PREFIX=`go list .`
DIR=$1

ARGS=""

for line in $(go list -deps $DIR); do
  if [[ "$line" == "$PREFIX"* ]]; then
    ARGS="$ARGS -w ${line/#$PREFIX/.}"
  fi
done

set -x
nodemon ${ARGS} --ext go --exec gotestsum --format dots $DIR