cibuildwheel CI failures from pip `WARNING: Skipping page https://pypi.org/simple/xxx/ because the GET request got Content-Type: Unknown`

Having seen more of this recently I investigated a little

WARNING: Skipping page https://pypi.org/simple/xxx/ because the GET request got Content-Type: Unknown ones.

My assumption is that it's this bug https://github.com/pypa/pip/issues/5345, related to pip concurrently accessing its cache - we do run tests in parallel.

This bug appears to be fixed in pip. However, we are still seeing it in cibuildwheel's CI - I'm not sure why that is, but ~it might be because we use an old version of pip to upgrade itself, and the old version of pip has this bug~ this doesn't appear to be the case!

Let's gather information about these failures here. Specifically, if we can figure out the version of pip that's outputting the error , that would be good.

Sep 04 '22 17:09 joerick

Here's an example failure - this one is from virtualenv, when it creates a new virtualenv it installs pip

Traceback (most recent call last):
  File "C:\Users\appveyor\AppData\Local\Temp\1\cibw-run-kny60zbz\cp37-win_amd64\build\venv\lib\site-packages\virtualenv\seed\embed\via_app_data\via_app_data.py", line 92, in _get
    env=self.env,
  File "C:\Users\appveyor\AppData\Local\Temp\1\cibw-run-kny60zbz\cp37-win_amd64\build\venv\lib\site-packages\virtualenv\seed\wheels\acquire.py", line 33, in get_wheel
    env=env,
  File "C:\Users\appveyor\AppData\Local\Temp\1\cibw-run-kny60zbz\cp37-win_amd64\build\venv\lib\site-packages\virtualenv\seed\wheels\acquire.py", line 67, in download_wheel
    raise subprocess.CalledProcessError(process.returncode, cmd, **kwargs)
subprocess.CalledProcessError: Command '['C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\cibw-run-kny60zbz\\cp37-win_amd64\\build\\venv\\Scripts\\python.exe', '-m', 'pip', 'download', '--progress-bar', 'off', '--disable-pip-version-check', '--only-binary=:all:', '--no-deps', '--python-version', '3.7', '-d', 'C:\\Users\\appveyor\\AppData\\Local\\pypa\\virtualenv\\wheel\\house', 'pip==20.0.2']' returned non-zero exit status 1.
ERROR:root:failed to download pip version 20.0.2, pip download exit code 1
The conflict is caused by:
    The user requested pip==20.0.2
    The user requested (constraint) pip==20.0.2
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
WARNING: Skipping page https://pypi.org/simple/pip/ because the GET request got Content-Type: .The only supported Content-Type is text/html
ERROR: Cannot install pip==20.0.2 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
Error: Command ['python', '-m', 'build', 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\pytest-of-appveyor\\pytest-1\\popen-gw1\\test_dependency_constraints_fi0\\project', '--wheel', '--outdir=C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\cibw-run-kny60zbz\\cp37-win_amd64\\built_wheel', '--config-setting='] failed with code 1. None

https://ci.appveyor.com/project/joerick/cibuildwheel/builds/44674131/job/rldlbep1xfn35y48#L413

Sep 04 '22 20:09 joerick

Here's another. This one is on the 'Installing build tools…' step

+ python -m pip install --upgrade pip -c /Users/appveyor/projects/cibuildwheel/cibuildwheel/resources/constraints-python39.txt
Requirement already satisfied: pip in ./lib/python3.9/site-packages (22.2.2)
+ which pip
/private/var/folders/zc/w54y8dpx71z33yjn8gmrlq3w0000gn/T/cibw-run-552sih0n/cp39-macosx_x86_64/build/venv/bin/pip
+ pip --version
pip 22.2.2 from /private/var/folders/zc/w54y8dpx71z33yjn8gmrlq3w0000gn/T/cibw-run-552sih0n/cp39-macosx_x86_64/build/venv/lib/python3.9/site-packages/pip (python 3.9)
+ which pip
+ which python
/private/var/folders/zc/w54y8dpx71z33yjn8gmrlq3w0000gn/T/cibw-run-552sih0n/cp39-macosx_x86_64/build/venv/bin/python
+ python --version
Python 3.9.13
+ which python

Installing build tools...
+ pip install --upgrade setuptools wheel delocate -c /Users/appveyor/projects/cibuildwheel/cibuildwheel/resources/constraints-python39.txt
Requirement already satisfied: setuptools in /private/var/folders/zc/w54y8dpx71z33yjn8gmrlq3w0000gn/T/cibw-run-552sih0n/cp39-macosx_x86_64/build/venv/lib/python3.9/site-packages (65.3.0)
Requirement already satisfied: wheel in /private/var/folders/zc/w54y8dpx71z33yjn8gmrlq3w0000gn/T/cibw-run-552sih0n/cp39-macosx_x86_64/build/venv/lib/python3.9/site-packages (0.37.1)
The conflict is caused by:
    The user requested delocate
    The user requested (constraint) delocate==0.10.2
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
                                                              ✕ 1.47s
------------------------------------------------------------------------------------------------------------- Captured stderr call -------------------------------------------------------------------------------------------------------------
WARNING: Skipping page https://pypi.org/simple/delocate/ because the GET request got Content-Type: Unknown. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/html
ERROR: Cannot install delocate because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
Error: Command ['pip', 'install', '--upgrade', 'setuptools', 'wheel', 'delocate', '-c', '/Users/appveyor/projects/cibuildwheel/cibuildwheel/resources/constraints-python39.txt'] failed with code 1. None

https://ci.appveyor.com/project/joerick/cibuildwheel/builds/44674131/job/2hmybiom075cs79x

The output is a bit jumbled, but interestingly, this failure appears to happen with pip version 22.2.2.

Sep 04 '22 20:09 joerick

Here's another, this time on Windows

  ERROR: Command errored out with exit status 1:

   command: 'C:\Users\runneradmin\AppData\Local\Temp\cibw-run-2ht01rzu\cp36-win32\build\venv\Scripts\python.exe' 'C:\Users\runneradmin\AppData\Local\Temp\cibw-run-2ht01rzu\cp36-win32\build\venv\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-jlszcw3c\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools >= 42' 'setuptools_scm[toml]>=4.1.2' wheel requests==2.23.0

///snip...///

  WARNING: Skipping page https://pypi.org/simple/setuptools/ because the GET request got Content-Type: .The only supported Content-Type is text/html

  ERROR: Could not find a version that satisfies the requirement setuptools>=42 (from versions: none)

  ERROR: No matching distribution found for setuptools>=42

https://github.com/pypa/cibuildwheel/actions/runs/3073026755/jobs/4964898440#step:9:184

Looks to be pip 21.3.1 on Python 3.6.

Sep 17 '22 13:09 joerick

Been doing some investigation of this over in https://github.com/joerick/pip-concurrency-debug .

My current theory is that it's an issue with concurrent cache access by pip - specifically concurrent cache access between pip<22.2 and pip>=22.2. I'm guessing there was a change to the cache at that point, and the bug happens (very occasionally) when versions before/after this change are reading/writing the same cache files.

Here's a test failure https://github.com/joerick/pip-concurrency-debug/actions/runs/3208382927/jobs/5244175446#step:5:24

Off the top of my head, I'm not totally sure why cibuildwheel doesn't just use 22.2.2 everywhere. virtualenv seeds environments with a vendored 22.2.2 afaik. Ah- could it be Python 3.6? Pip 22.2.2 isn't compatible with 3.6.

Oct 08 '22 00:10 joerick

I'm feeling more confident that this is the issue. We started noticing this in August, and we started using pip 22.2 at the end of July.

To be clear, I don't think this is an issue with pip 22.2 - it's an issue when using pip<22.2 and pip>22.2 simultaneously. The only case that we use pip<22.2 is in Python 3.6.

Solutions-

specify a separate PIP_CACHE_DIR for Python 3.6
drop Python 3.6 (it's already 9+ months past EOL)

I'd be in favour of dropping 3.6 myself - people can always do CIBW_BUILD=cp36-* pipx run cibuildwheel==2.10.2 if they need.

Oct 08 '22 18:10 joerick

I have found a Pip thread tracking this issue and reported my findings there too.

Oct 08 '22 19:10 joerick

I thought we planned to drop 3.6 when Ubuntu 18.04 hits EOL & manylinux drops it - I believe next April? Specifically, I really like the idea of always matching manylinux. I don't care too much about 3.6 itself, I just like matching manylinux.

We could, in the interim, possibly split out the 3.6 testing, and/or reduce it quite a bit (to just 1-2 jobs). Oh, or the PIP_CACHE_DIR idea too.

Oct 08 '22 19:10 henryiii

There's no definitive plan to drop 3.6 in manylinux yet. The only thing hinted in https://github.com/pypa/manylinux/issues/1260 is that it will not happen before Ubuntu 18.04 end of active support (2023-04) but probably before Ubuntu 18.04 end of security support (2028-04).

3.6 packages are still built by 30% of projects & consumed by 16% of downloads.

We could, in the interim, possibly split out the 3.6 testing, and/or reduce it quite a bit (to just 1-2 jobs). Oh, or the PIP_CACHE_DIR idea too.

yeah, there should be no issue at all building 3.6 for cibuildwheel users given its a concurrency thing with parallel tests. I don't mind reducing test coverage for 3.6, I think we don't have any specific workarounds there so let's identify 1 or 2 tests jobs and apply the PIP_CACHE_DIR there. There's immediately 1 test that comes to mind that would require the PIP_CACHE_DIR even if we do drop 3.6: test_dependency_version

Oct 09 '22 10:10 mayeut