Hatch has too many dependencies

[orsinium]

Here is what needs to be installed to install hatch:

anyio==3.7.1
certifi==2023.7.22
cffi==1.15.1
click==8.1.7
cryptography==41.0.3
distlib==0.3.7
editables==0.5
filelock==3.12.2
h11==0.14.0
hatchling==1.18.0
httpcore==0.17.3
httpx==0.24.1
hyperlink==21.0.0
idna==3.4
importlib-metadata==6.8.0
jaraco-classes==3.3.0
jeepney==0.8.0
keyring==24.2.0
markdown-it-py==3.0.0
mdurl==0.1.2
more-itertools==10.1.0
packaging==23.1
pathspec==0.11.2
pexpect==4.8.0
platformdirs==3.10.0
pluggy==1.2.0
ptyprocess==0.7.0
pycparser==2.21
pygments==2.16.1
rich==13.5.2
secretstorage==3.3.3
shellingham==1.5.3
sniffio==1.3.0
tomli-w==1.0.0
tomlkit==0.12.1
trove-classifiers==2023.8.7
userpath==1.9.0
virtualenv==20.24.3
zipp==3.16.2

That's a lot, and most of it isn't needed if you don't use a very specific feature or are just conveniences that can be rewritten:

1. hyperlink is only used to change host and path in the index URL and can be replaced by urllib from stdlib.
2. keyring is only used for package publishing which isn't needed for internal projects or projects that use twine for it.
3. pexpect is used only on UNIX machines and only when activating venv. And I never needed to activate venv because hatch provides run to run anything in the correct environment.
4. shellingham is the same story, only needed for activating a shell.
5. platformdirs can be skipped on major OSes by hardcoding their user cache dir and user data dir detection logic (check existence of a few paths, check a few env vars, and that's it).
6. tomli-w is needed only for hatch new which is called only once in the project existence, not needed by the project users, and never needed on CI.
7. tomlkit and tomli-w have overlapping functionality (writing TOML files), pick one.
8. trove-classifiers is used to validate classifiers which is done by PyPI anyway. And it certainly not needed in hatchling, all validations should be done earlier, in hatch.
9. userpath seems to be unused.

Every dependency makes project installation more slow and complicated, packaging the project downstream convoluted, and has a negative effect on CI performance which affects all users and accumulates over time.

[ofek]

userpath will be used and tomli-w will be removed soon. otherwise I don't know what you would expect me to do here. can you explain your frustration in more detail please?

[orsinium]

Issues:

1. Longer times to install hatch -> slower CI builds in the base images without hatch preinstalled.
2. More points of failure that are out of hatch control. For example, there were issues with cryptography on Alpine when it moved to Rust. That means, in a similar scenario hatch installation would fail too.
3. More complicated installation. Hatch is a package manager and will be installed in the global Python environment more often than you'd expect, not everyone uses pipx. That means, that if its dependency constraints conflict with something in the same environment, users will have troubles.

Solutions (depend on a specific dependency, I outlined the best solution for each one in the original post):

1. Make dependencies optional, tell users to install hatch[shell] when they need shell support, hatch[upload] when they need support for uploading distributions, or hatch[all] as the default installation when dependencies don't matter.
2. Make OS-specific dependencies to be installed only on these OSes and skip on others using dependency markers.
3. "A little copying is better than a little dependency". If a dependency can be replaced by a few lines of code, do it.
4. Prefer stdlib whenever possible and prefer using already available dependency instead of bringing another one.

I don't expect anything from you specifically. Well, maybe to understand risks and issues with having lots of dependencies in a package manager, why pip vendors all of its dependencies, and why I had to solve a similar issue in dephell years ago (https://github.com/dephell/dephell/issues/347). Other than that, I'd be happy to contribute a solution to at least some of these, and perhaps other people would be interested in helping too.

[ofek]

userpath is used now https://github.com/pypa/hatch/pull/970

[dimaqq]

I'm a little sad that cffi is used, as it requires gcc in docker images which would otherwise be small, e.g. based of python:3.12.1-alpine3.19

[ofek]

cffi is not a dependency, what are you referring to?

[dimaqq]

the dep chain is hatch --> keyring --> something --> cffi basically pip install hatch in a blank virtualenv ends up pulling in cffi, or fails entirely if gcc is not available

[dimaqq]

I suppose the alternative is to install pre-built binaries, which kindof makes sense.

[orsinium]

@dimaqq is right. See the issue description above, it lists all dependencies, including transitive ones, that hatch brings. Back in August, it had 39 dependencies. Now the number might be higher, I haven't checked.

[ofek]

I would be interested in one of you providing a traceback. If a compiler is involved then likely what you are seeing is a transitive dependency that does not have a wheel on PyPI and during that build process CFFI is required.

Overall, I just need more information please.

[orsinium]

The path is hatch/keyring/SecretStorage/cryptography/cffi. Steps to reproduce with pipdeptree:

python3 -m venv .venv
.venv/bin/python -m pip install hatch
pipdeptree --python .venv/bin/python3

The tree:

hatch==1.9.3
├── click [required: >=8.0.6, installed: 8.1.7]
├── hatchling [required: >=1.21.0, installed: 1.21.1]
│   ├── editables [required: >=0.3, installed: 0.5]
│   ├── packaging [required: >=21.3, installed: 23.2]
│   ├── pathspec [required: >=0.10.1, installed: 0.12.1]
│   ├── pluggy [required: >=1.0.0, installed: 1.4.0]
│   ├── tomli [required: >=1.2.2, installed: 2.0.1]
│   └── trove-classifiers [required: Any, installed: 2024.1.8]
├── httpx [required: >=0.22.0, installed: 0.26.0]
│   ├── anyio [required: Any, installed: 4.2.0]
│   │   ├── exceptiongroup [required: >=1.0.2, installed: 1.2.0]
│   │   ├── idna [required: >=2.8, installed: 3.6]
│   │   ├── sniffio [required: >=1.1, installed: 1.3.0]
│   │   └── typing-extensions [required: >=4.1, installed: 4.9.0]
│   ├── certifi [required: Any, installed: 2023.11.17]
│   ├── httpcore [required: ==1.*, installed: 1.0.2]
│   │   ├── certifi [required: Any, installed: 2023.11.17]
│   │   └── h11 [required: >=0.13,<0.15, installed: 0.14.0]
│   ├── idna [required: Any, installed: 3.6]
│   └── sniffio [required: Any, installed: 1.3.0]
├── hyperlink [required: >=21.0.0, installed: 21.0.0]
│   └── idna [required: >=2.5, installed: 3.6]
├── keyring [required: >=23.5.0, installed: 24.3.0]
│   ├── importlib-metadata [required: >=4.11.4, installed: 7.0.1]
│   │   └── zipp [required: >=0.5, installed: 3.17.0]
│   ├── jaraco.classes [required: Any, installed: 3.3.0]
│   │   └── more-itertools [required: Any, installed: 10.2.0]
│   ├── jeepney [required: >=0.4.2, installed: 0.8.0]
│   └── SecretStorage [required: >=3.2, installed: 3.3.3]
│       ├── cryptography [required: >=2.0, installed: 42.0.1]
│       │   └── cffi [required: >=1.12, installed: 1.16.0]
│       │       └── pycparser [required: Any, installed: 2.21]
│       └── jeepney [required: >=0.6, installed: 0.8.0]
├── packaging [required: >=21.3, installed: 23.2]
├── pexpect [required: ~=4.8, installed: 4.9.0]
│   └── ptyprocess [required: >=0.5, installed: 0.7.0]
├── platformdirs [required: >=2.5.0, installed: 4.1.0]
├── rich [required: >=11.2.0, installed: 13.7.0]
│   ├── markdown-it-py [required: >=2.2.0, installed: 3.0.0]
│   │   └── mdurl [required: ~=0.1, installed: 0.1.2]
│   └── pygments [required: >=2.13.0,<3.0.0, installed: 2.17.2]
├── shellingham [required: >=1.4.0, installed: 1.5.4]
├── tomli-w [required: >=1.0, installed: 1.0.0]
├── tomlkit [required: >=0.11.1, installed: 0.12.3]
├── userpath [required: ~=1.7, installed: 1.9.1]
│   └── click [required: Any, installed: 8.1.7]
├── virtualenv [required: >=20.16.2, installed: 20.25.0]
│   ├── distlib [required: >=0.3.7,<1, installed: 0.3.8]
│   ├── filelock [required: >=3.12.2,<4, installed: 3.13.1]
│   └── platformdirs [required: >=3.9.1,<5, installed: 4.1.0]
└── zstandard [required: <1, installed: 0.22.0]
pip==22.0.2
setuptools==59.6.0

[ofek]

Cryptography has wheels for every platform, why are you building from scratch? Is this about builds for a distribution?

[dimaqq]

I my case, the setup is:

• mbp (m1)
• rancher desktop (+vz +moby -k8s)
• FROM python:3.12.1-alpine3.19

# in a container:
/ # uname -a
Linux b893d13c80ad 6.1.64-0-virt #1-Alpine SMP Wed, 29 Nov 2023 18:56:40 +0000 aarch64 Linux

To reproduce, docker run -it python:3.12.1-alpine3.19 pip install hatch > stdout 2> stderr

stdout.md

Excerpt:

Collecting cryptography>=2.0 (from SecretStorage>=3.2->keyring>=23.5.0->hatch)
  Obtaining dependency information for cryptography>=2.0 from https://files.pythonhosted.org/packages/d8/41/1e2cfc14cdae6ad0b5c6677e2cb03af2a6e01c05a72d5b3fddf693b26f3d/cryptography-42.0.1-cp39-abi3-musllinux_1_2_aarch64.whl.metadata
  Downloading cryptography-42.0.1-cp39-abi3-musllinux_1_2_aarch64.whl.metadata (5.3 kB)
Collecting more-itertools (from jaraco.classes->keyring>=23.5.0->hatch)
  Obtaining dependency information for more-itertools from https://files.pythonhosted.org/packages/50/e2/8e10e465ee3987bb7c9ab69efb91d867d93959095f4807db102d07995d94/more_itertools-10.2.0-py3-none-any.whl.metadata
  Downloading more_itertools-10.2.0-py3-none-any.whl.metadata (34 kB)
Collecting cffi>=1.12 (from cryptography>=2.0->SecretStorage>=3.2->keyring>=23.5.0->hatch)
  Downloading cffi-1.16.0.tar.gz (512 kB)

And then it proceeds trying to build cffi from source.

[dimaqq]

My guess is that cryptography depends on cffi: https://github.com/pyca/cryptography/blob/285ebed5e49bfd15b1a37cdbc8d85ddddd555f51/pyproject.toml#L47-L50 and cffi doesn't have wheels for this specific combination.

• has cp312-musllinux_1_1_x86_64
• has cp312-macosx_11_0_arm64
• has many others
• does not have cp3*-musl*-arm64

[ofek]

Nice call, that appears to be exactly what's happening! Please follow this issue for updates: https://github.com/python-cffi/cffi/issues/41