Move to hatchling for building the package & use hatch for envs, scripts, matrix tests.

[BSchilperoort]

Currently, the template is set up to build the package with setuptools. ~~However, the current advise on packaging.python.org is to use hatchling. (related to #318)~~ Edit: The default option is hatch. No specific option is recommended.

This would also be an opportunity to use hatch as project manager. Hatch uses hatchling to build the package in a separate virtual environment, has environment management with scripts, and allows much easier publishing to pypi. Additionally, hatch is maintained by the Python Packaging Authority.

Hatch would allow us to:

• Make a more clean repository, with fewer configuration files in the main directory (hatch is configured in pyproject.toml)

• Define environments, such as a specific environment for building documentation, with the documentation dependencies contained only in there.

• Define script belonging to a certain environment, for example hatch run docs:build to build the documentation.

• Define scripts for linting, to replace prospector. With a single command hatch run lint, you can (for example) run black, isort, flake8, mypy, pydoctest, etc.

• Use matrix tests for different python versions. This could replace tox.

Together with @Peter9192 I have updated era5cli to use hatch, if you want to see a specific use case. The CI is set up to use the same hatch commands that developers do.

[Peter9192]

Hatch also provides template configuration: https://hatch.pypa.io/latest/config/project-templates/ so we could set e.g. standard license to apache.

[bouweandela]

However, the current advise on packaging.python.org is to use hatchling.

On the linked documentation page, I see that they use hatchling as the default choice for the tutorial, but I see no advice to prefer this over any other build system. Could you give more information on that? Or did I miss it somehow?

[BSchilperoort]

You're correct, Bouwe. I'll modify the top post.

For additional discussion, I have found this nice overview of python packaging options: https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/#tooling-proliferation-and-the-python-package-authority That article also raises an important point of the (current) bus factor of hatch essentially being 1.

The lack of C-extension support could also be a deal-breaker for some. But I don't know how common that is in the center.

[egpbos]

Yeah, C-extension support is pretty important. There are a couple of projects that do things with pybind11 for instance.

[ofek]

Hatchling does support custom build hooks so one could still perform the logic of building the extension modules

[bouweandela]

I'm not sure if it's worth the effort of changing the template if there is no clear recommendation. Many of the things listed as advantages of using hatch can also be done easily using other tools in common use. For example

• Publishing to PyPI and running matrix tests are easy to set up using GitHub actions
• Running multiple tools for linting and formatting is a task that is typically done using pre-commit. Even if you could set up a hatch command to do this for you, you would still want to use pre-commit and then what's the advantage of maintaining a duplicate configuration?
• Defining extra dependencies for building the documentation can be done using extras
• Configuration in pyproject.toml is something that is supported by many tools, including the currently used setuptools

[bouweandela]

Related to this, we may want to keep an eye on uv

[ofek]

The next minor release of Hatch will have an option to use that behind the scenes in place of pip and virtualenv.