Specify dependency groups in a way that doesn't require poetry

[mfisher87]

"Dependency groups" are a poetry-specific feature that's different from PEP621 "optional dependencies", aka "extras". To make things compatible with other ways of dependency management, switch to PEP621 optional dependencies, but specified using the Poetry configuration aliases to retain support for poetry. While the Poetry "dependency groups" functionality may be more aimed at development workflows, it's not compatible with other tools and in my experience the dominant convention in the ecosystem is to use optional dependencies. As @itcarroll points out below, Hatch supports a similar concept called "environment dependencies" and specifies them with a different tool-specific configuration key. PDM also calls it something different, "dev dependencies", and uses another proprietary config table. :sweat:

I really don't like the way poetry requires optional dependencies to be specified. It requires duplication.

Thoughts? I don't want to add all this duplication to our codebase. Can we go forward with replacing Poetry with good ol' setuptools or maybe hatchling for builds and switch the optional dependency groups to the standard format? Anything PEP621 compatible would be a step forward IMO. I can take that on as part of this PR. The PEP621 example shows our use case, test dependencies, implemented in the conventional way.

Closes #734 Closes #619

---

📚 Documentation preview 📚: https://earthaccess--733.org.readthedocs.build/en/733/

[itcarroll]

While the Poetry "dependency groups" functionality may be more aimed at development workflows, it's not compatible with other tools and in my experience the dominant convention in the ecosystem is to use optional dependencies.

I have been exploring hatch w/ hatchling, and it is both PEP compliant and also (like Poetry) avoids using extras (a.k.a optional-dependencies) for managing development environments. Information about extras is built into the distribution metadata and primarily provides extra packages users might want (i.e. xarray[io] if you want netCDF4, etc.). Hatch (and Poetry) allow separate ways of providing packages for and setting up development environments; information not included in distribution metadata. I haven't been compelled to duplicate package lists for simple cases with Hatch. Note that the current potential downside with Hatch is a lack of lock files.

[mfisher87]

Just to clarify, I'm for including these optional dependency groups in the package metadata, as that's in my experience a very common method and is compatible with all environment / package managers, and as far as I know there is no standard way to specify dependency groups that do not go in the package metadata.

I'd like to avoid pros and cons of environment / package managers here (i.e. Hatch, Poetry), as we can isolate that concern from our build tooling and configuration. Regardless of whether we decide down the road to document / encourage Hatch or Poetry or something else for package management, if our build configuration is PEP621 compliant, contributors can choose any tool.

[mfisher87]

To make things more confusing, PDM has "dev dependencies", their own uniquely named version of Poetry's "dependency groups" and Hatch's "environment dependencies". Wow! The "same" thing three times with three names and three config keys.

I'd really like to treat this Python environment management feature as non-standard and avoid it until there's a unified interface, which I'm sure there's already an open PEP for :laughing:

[mfisher87]

Found it: https://peps.python.org/pep-0735/

[mfisher87]

Thanks, Chuck! Good point, I do imagine I missed some stuff there.

TODO:

• [x] Ensure docs are up-to-date with dependency management changes

[mfisher87]

@chuckwondo @danielfromearth I committed the changes we were working on in the call (24a5191), and clearly they are no longer poetry compatible :) I didn't have time to debug yet!

[danielfromearth]

So, where do we want to go from here? If there's no way to remove the redundant dependency specifications, then get this working with another manager, like hatch, and then drop the poetry usage?

[mfisher87]

Personally, dropping poetry in favor of hatch, uv, pip, and all the other standards-compliant dependency managers feels like the right way forward. Do we have consensus on that? I'm not sure :) #761

[itcarroll]

dropping poetry in favor of hatch, uv, pip

Reminder that hatch does not support lock files (waiting for a standard, I believe).

So, where do we want to go from here?

An option is to wait until poetry supports the project table (https://github.com/python-poetry/poetry/issues/9448). On that ... I am confused by the inclusion of the project.optional-dependencies table in this PR, because AFAIK poetry does not support the project table at all. Is it included here to demonstrate that failure (maybe mark as "Draft" in that case)?

[mfisher87]

I am confused by the inclusion of the project.optional-dependencies table in this PR, because AFAIK poetry does not support the project table at all

The most recent commit was made without fully understanding that fact in an effort to test what would happen :) If we revert that commit, it goes back to the poetry-specific specification. But... is that what we want? :shrug: I find the poetry way much harder to read.

Reminder that hatch does not support lock files (waiting for a standard, I believe).

I don't think lack of a lockfile is a dealbreaker, personally, but we could use uv lock.

[danielfromearth]

hmm, hadn't seen uv yet, but that looks quite promising!

It does look like the particular poetry PR for the project section dependencies (https://github.com/python-poetry/poetry/pull/9135) might be close to being finalized (all tests are passing just marked ready-for-review 4 days ago).

[mfisher87]

It does look like the particular poetry PR for the project section dependencies (python-poetry/poetry#9135) might be close to being finalized (all tests are passing just marked ready-for-review 4 days ago).

While it would be nice to not have to do the work to rip out poetry given this change :point_up: I still think we should rip out poetry. I see poetry as purely a source of complexity. E.g. why do we need ^ in Python? We have a compatible release specifier (~=), but we want it to feel more like Javascript? :shrug:

Anthony Sottile (pre-commit): why I will never use python-poetry

[mfisher87]

Thanks all for your help! :bow: