Separate doctests into a separate Nox session and GitHub workflow?

[namurphy]

I'm wondering if it would be worthwhile to have a dedicated Nox session and GitHub workflow to check that doctests are working. At the moment I think it would be beneficial to do this, but it'd be worth discussing this further before attempting to implement this.

Advantages:

• We'd be more likely to notice if the doctests are not being run (see #2834)
• We could more reliably ensure that doctests are not counted when checking code coverage (so that we can identify lines of code that the test suite itself does not cover)
• We could shorten the time it takes to run the test suite. I'm not sure if the improvement will be negligible or noticeable, but I'd guess maybe a few seconds?
• We could have somewhat more computationally intensive examples without lengthening the overall time it takes tests to run.
• With pytest-doctestplus enabled, we could have a Nox session that would check and optionally autofix docstrings. When doctests fail, we could print out the command to run to perform autofixes.
• We could more easily isolate problems with doctests.

Disadvantages:

• When running Nox sessions locally, we'd have to run more than one to run test.
• We'd length the list of checks that get run in each PR.

[lemmatum]

Given the move to uv with #2959 is there a reason to use nox/tox? It seems like a matrix job in Github Actions plus uv should be able to run all the tests, unless I'm missing something?

[namurphy]

That's a good question! While uv really does improve the situation, Nox makes it straightforward for us to run the same checks or tasks with the same setup both locally and on CI. Nox really helps when running a test requires a few lines of setup or when a command requires multiple options which are hard to keep track of or that could potentially change in the future. Personally I've found it works really well to use uv and Nox in conjunction with each other, including with uv as a backend to Nox. Nox defines the workflows that get performed, and uv run or uvx is used to execute particular tasks within those workflows.