Documentation: Explain or link to info about using the package for tests

[mgrazebrook]

Python Packaging Projects says:

"tests/ is a placeholder for test files. Leave it empty for now."

And indeed this is something I forget and have to look up every time :anguished:. It would be nice to include a good reference or even a short section giving the basics.

Things I'd find especially useful would be a reminder of how to set up the config files so I can develop in editable mode and run tests against my installed package.

This is especially useful because the link you do have to Python documentation for packages and modules doesn't mention this either: I don't think there's any direct link from Python Packaging Projects to anything helpful in setting up ones tests to use the package. For me, it's a gap.

[bhrutledge]

@mgrazebrook I agree, and it's been in my personal backlog for awhile: https://github.com/pypa/packaging.python.org/issues/619. Is it okay if I close this issue as a duplicate?

[mgrazebrook]

I see value in combining them, though I feel I'm commenting from a different angle. Your ticket is about the layout of tests. Mine is about the config files to support them.

I skimmed the older ticket - I wonder if 'controversial' is delaying action. IMO, pick your own favourite, note the controversy - it's not the one true way - perhaps with a link and maybe split the task into a quick fix and a longer term polishing. For me, I spend hour trying to get this right each time I do it, so something adequate would be of real benefit.

Let me know if I can help, perhaps as an intermediate-level reviewer.

[bhrutledge]

Have a look at https://github.com/bhrutledge/example-pkg-bhrutledge; that's my sandbox for the packaging tutorial, and I was planning to use its test setup as the reference point. Are there any changes you would make there?

[mgrazebrook]

These suggestions aren't necessarily better than what you have but may give ideas.

Maybe run pytest as

python -m pytest

This ensures you pick up your environment. Personally I like to run pytest with extra options:

python -m pytest --cov --cov-report term-missing

However that's probably too elaborate.

I think the packages you need might be best installed through a commented config file, maybe requirements.txt but I'm not sure that's the 'modern' way of doing it.

You're proposing pip install -e which I definitely like. However as that's only for development use, maybe - if one moves the packages to install to config files - one could have dev and prod config files (e.g. requirements-dev.txt which imports the prod config and includes '-e .'

I like your choice to promote using a src directory.

Do you think one should promote using init.py under tests as noted here: Good Integration Practices?

Just remember - I'm not that good at packaging and setting out the structure. So my ideas may not be best pratcice.

[henryiii]

This page (https://scikit-hep.org/developer/pytest) along with the related pages cover pytest, might be worth checking out.

pip install -e . doesn't mean you have to install development requirements. You might want to test without all your dev requirements (with minimal test requirements at least, I'd use -e .[test]), but you still want the reported filenames to point at your real files, not some place in a virtual environment. requirements.txt is really more about locked environments, IMO, not a place to put regular requirements. Extras for most cases, requirements.in + piptools, or using a tool like PDM, Poetry, or Hatch.

If you expect arguments always, I'd include them in your config, or in your nox/tox file - a user shouldn't have to add flags to use it. And adding coverage reporting is an "extra" that slows down your test suite a lot, so a normal user likely doesn't need it.

Adding __init__.py is something I'm not sure of. I'd strongly avoid it in the past unless you need to import inside your test suite (you rarely do, a fixture in contest.py is often better). You tests are not really packages, and you very much do not want them to accidentally get packaged as a package in your wheels! If you do need to import inside a test suite, in future versions of PyTest, you might be required to have an __init__.py (or manually define paths, etc) due to the move to importlib. I've not had success with it, though, in macOS and Windows, so I've so far avoided importing if I am using the importlib loader.

[mgrazebrook]

Thanks for the advice. I guess part of the challenge is that there are so many ways to approach the configuration. poetry, ... as you say. Isn't Pipfile.lock' the best practice way to capture precise dependencies? Documenting the best practice way to specify dependencies without giving versions might be nice - something which encourages people to use the most recent versions while developing but nail down safe configurations on release. Recent versions are arguably better for security? Maybe a best practice way to provide version ranges could be part of the default docs - though I don't know how to do that. It's annoying when you can't use the most recent version of a package because some other package is being overly restrictive.

You're right about pip install -e, I guess it's always better.

[abravalheri]

This might be relevant here too: https://blog.ganssle.io/articles/2019/08/test-as-installed.html

[henryiii]

because some other package is being overly restrictive.

I've written ~10,000 words on that... https://iscinumpy.dev/post/bound-version-constraints/ There are several other similar linked posts from others of us there too. ;)

I don't think there's a clear "best practice" for capturing precise dependencies if you are looking for a specific tool. The practice itself is recommended and well defined, but there are several ways to do it. Pipfile.lock is pipenv, which is okay but not really highly recommended - the person maintaining it is also the author of PDM, which is better, I think. Piptools is good if you want something close to classic tooling (that is, you want separation of packaging and dependencies). Etc.