hatch Several tests related to Python distributions fail on Fedora Linux on non-x86

I’m testing an update from python-hatchling-1.21.1 to python-hatchling-1.25.0 and hatch-1.9.7 to hatch-1.12.0 in Fedora Rawhide. (This has been stalled for a few months by the need to package uv, but that’s done now.)

All tests pass on x86_64.

On ppc64le:

FAILED tests/python/test_resolve.py::TestDistributionPaths::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionVersions::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='ppc64le' abi='gnu' variant=''

On aarch64:

FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='aarch64' abi='gnu' variant=''

On s390x:

FAILED tests/python/test_core.py::test_custom_source[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionPaths::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionVersions::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''

(We don’t build noarch packages on i386/i686 anymore, so I didn’t test it.)

This resembles https://github.com/pypa/hatch/issues/1145.

Aug 08 '24 18:08 musicinmybrain

We do have an older python-virtualenv-20.21.1 with some backported patches, but currently without https://github.com/pypa/virtualenv/pull/2709 or https://github.com/pypa/virtualenv/pull/2712 (https://github.com/pypa/hatch/issues/1395#issuecomment-2276294348). I’m testing with the lower bound on the version of virtualenv removed. I didn’t think that should be affecting this, but I thought it was worth mentioning. To be certain, I tried reproducing this in a Fedora Rawhide chroot on ppc64le using qemu-user-static emulation. The packages gcc cargo python3-devel openssl-devel openssl-devel-engine are needed because cryptography still has to be built from source on Python 3.13. I can’t use hatch test, or install xdist[psutil], because psutil also still has to be built from source, and it looks for gcc in the wrong place (/usr/bin/powerpc64le-linux-gnu-gcc), as if it were being cross-compiled.

$ mock -r fedora-rawhide-ppc64le --clean
$ mock -r fedora-rawhide-ppc64le -i  git-core cargo python3-devel openssl-devel openssl-devel-engine
$ mock -r fedora-rawhide-ppc64le --shell --enable-network
# git clone https://github.com/pypa/hatch.git
# cd hatch
# git checkout hatch-v1.12.0
# python3 -m venv _e
# . _e/bin/activate
(_e) # python --version
Python 3.13.0rc1
(_e) # pip install -e .
(_e) # pip install pytest pytest-mock pytest-randomly pytest-rerunfailures pytest-xdist editables
(_e) # python -m pytest -n auto
[…]
FAILED tests/python/test_core.py::test_custom_source[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionPaths::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionVersions::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/cli/test/test_test.py::TestCustomScripts::test_single - AssertionError: assert [call('test h..., shell=True)] == [call('test h..., shell=True)]
FAILED tests/cli/run/test_run.py::TestScriptRunner::test_python_version_constraint - AssertionError: assert 'Creating env...ependencies\n' == 'Creating env...ependencies\n'
FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='ppc64le' abi='gnu' variant=''
ERROR tests/utils/test_auth.py::test_pypirc

This is slightly more failures than I reported – some, like TestCustomScripts::test_single, appear to be artifacts of how I’m running the tests here, and some perhaps are because this test had network access, unlike Fedora package builds – but it’s still safe to say that this basically reproduces the issue. Since this test used an unpatched hatch/hatchling and dependencies from PyPI and respected all version bounds, it seems safe to say that Fedora’s python-virtualenv-20.21.1 is not involved.

Aug 08 '24 18:08 musicinmybrain

Although Fedora 40 won’t get a Hatch update, I repeated the experiment from https://github.com/pypa/hatch/issues/1670#issuecomment-2276422947 in a Fedora 40 ppc64le chroot, where the system Python is 3.12. I still have to compile cryptography from source. I thought that would have been a Python 3.13 thing, but instead it turns out that cryptography only releases binary wheels for x86_64 and aarch64.

$ mock -r fedora-40-ppc64le --clean
$ mock -r fedora-40-ppc64le -i git-core cargo python3-devel openssl-devel
$ mock -r fedora-40-ppc64le --shell --enable-network
# git clone https://github.com/pypa/hatch.git
# cd hatch
# git checkout hatch-v1.12.0
# python3 -m venv _e
# . _e/bin/activate
(_e) # python --version
Python 3.12.4
(_e) # pip install -e .
(_e) # hatch test
[…]
=========================================================================================== short test summary info ============================================================================================
FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionVersions::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionPaths::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='ppc64le' abi='gnu' variant=''
=========================================================================== 12 failed, 1996 passed, 69 skipped in 582.37s (0:09:42) ============================================================================

Aug 08 '24 19:08 musicinmybrain

Repeating the Fedora 40 / Python 3.12 test from https://github.com/pypa/hatch/issues/1670#issuecomment-2276490421 for s390x gives:

=========================================================================================== short test summary info ============================================================================================
FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_custom_source[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.8] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.8' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy2.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy2.7' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy3.9] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.9' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[pypy3.10] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionVersions::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''
FAILED tests/python/test_resolve.py::TestDistributionPaths::test_pypy_custom - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='pypy3.10' system='linux' arch='s390x' abi='gnu' variant=''
=========================================================================== 12 failed, 1996 passed, 69 skipped in 634.55s (0:10:34) ============================================================================

…and aarch64 gives:

=========================================================================================== short test summary info ============================================================================================
FAILED tests/python/test_core.py::test_custom_source[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='aarch64' abi='gnu' variant=''
FAILED tests/python/test_core.py::test_installation[3.7] - hatch.errors.PythonDistributionResolutionError: Could not find a default source for name='3.7' system='linux' arch='aarch64' abi='gnu' variant=''
=========================================================================== 2 failed, 2006 passed, 69 skipped in 4242.38s (1:10:42) ============================================================================

Aug 09 '24 14:08 musicinmybrain

What would be the fix that you would recommend?

Oct 15 '25 13:10 ofek

What would be the fix that you would recommend?

I’m not sure. I suppose it depends on what you are actually trying to test.

We could follow the general example of

https://github.com/pypa/hatch/blob/f16441ff03259a7c8f47eb6d03bd406213694f11/tests/python/test_core.py#L14-L15

and maintain in tests/python/test_core.py a list of (platform, current_arch, name) distribution tuples that don’t exist in hatch.python.distributions.DISTRIBUTIONS and should be skipped. This feels kind of fussy, and it’s a little awkward to maintain a copy of this “data” in the tests, but it ought to work.

One could do this automatically by consulting DISTRIBUTIONS in the test, but then are we really testing anything?

Happy to entertain other (better?) suggestions and/or work up a PR.

Oct 16 '25 05:10 musicinmybrain

Here’s what I’m currently doing in Fedora to work around this:

# https://github.com/pypa/hatch/issues/1670
# hatch.errors.PythonDistributionResolutionError: Could not find a default
# source for name='${param}' system='linux' arch='${arch}' abi='gnu' variant=''
#
# Since we cannot use %%ifarch in a noarch package, we skip tests that fail on
# *any* architecture, even if we happen to be building on an architecture where
# they pass. The largest set of failing distributions are, for ppc64le/s390x:
missing_dists='3.7 3.8 pypy2.7 pypy3.9 pypy3.10 pypy3.11'
# For aarch64, missing_dists='3.7', and for x86_64, missing_dists=''.
for param in ${missing_dists}
do
  k="${k-}${k+ and }not test_custom_source[${param}]"
  # Requires network (but does fail when it is enabled)
  k="${k-}${k+ and }not test_installation[${param}]"
done
# Also, for ppc64le and s390x:
k="${k-}${k+ and }not (TestDistributionPaths and test_pypy_custom)"
k="${k-}${k+ and }not (TestDistributionVersions and test_pypy_custom)"

%pytest -k "${k-}" ${ignore-} -vv

Oct 16 '25 05:10 musicinmybrain

Several tests related to Python distributions fail on Fedora Linux on non-x86_64 architectures in Hatch 1.12.0