pip pinning is incomplete

[Habbie]

#10776 introduced hashed pinning for the full pip requirements of pdns/docs. Or so we thought. This morning the build broke, which should be impossible.

Yesterday, Cython 3.0.0 was released. Cython 3's incompatibility with PyYAML was long known, and PyYAML 6.0.1 (released today) has a workaround.

Indeed, bumping pyyaml to 6.0.1 in our requirements unbreaks the build. However, all of this suggest that Cython is being pulled in unpinned, and it is:

/pdns/docs # .venv/bin/pip install -v  -r requirements.txt 
....
Collecting pyyaml==6.0 (from -r requirements.txt (line 228))
  Using cached PyYAML-6.0.tar.gz (124 kB)
  Running command pip subprocess to install build dependencies
  Collecting setuptools
    Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/c7/42/be1c7bbdd83e1bfb160c94b9cafd8e25efc7400346cf7ccdbdb452c467fa/setuptools-68.0.0-py3-none-any.whl.metadata
    Using cached setuptools-68.0.0-py3-none-any.whl.metadata (6.4 kB)
  Collecting wheel
    Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
  Collecting Cython
    Obtaining dependency information for Cython from https://files.pythonhosted.org/packages/88/90/e94b3dc8d0a988b4c8a98b5058ff57d677588b6dba657602c1c958192bf3/Cython-3.0.0-cp311-cp311-musllinux_1_1_x86_64.whl.metadata
    Using cached Cython-3.0.0-cp311-cp311-musllinux_1_1_x86_64.whl.metadata (3.1 kB)
  Using cached setuptools-68.0.0-py3-none-any.whl (804 kB)
  Using cached Cython-3.0.0-cp311-cp311-musllinux_1_1_x86_64.whl (3.6 MB)
  Installing collected packages: wheel, setuptools, Cython
  Successfully installed Cython-3.0.0 setuptools-68.0.0 wheel-0.40.0

Indeed, an unpinned dep, that is downloaded.

https://pip.pypa.io/en/stable/topics/repeatable-installs/ has a Warning at the bottom, linking to https://pip.pypa.io/en/stable/reference/build-system/#controlling-setup-requires, suggesting we can solve the problem there.

Experiment:

/pdns/docs # cat ~/.pydistutils.cfg 
[easy_install]
index_url = https://downloads.powerdns.com/tmp/

(I picked a URL that I control, that definitely does not contain a valid pip feed).

Fresh venv, with pyyaml dep bumped to 6.0.1:

/pdns/docs # rm -rf /root/.cache
/pdns/docs # .venv/bin/pip install -v  -r requirements.txt 
...

• sphinxcontrib-openapi has a hidden dep on setuptools_scm>=1.15
• Cython is still pulled in, unpinned
• sphinxcontrib-fulltoc has a hidden dep on pbr

Installing pbr and seutptools_scm by hand makes the build work, but Cython remains problematic.

[Habbie]

Related observation: .venv/bin/pip install -U pip setuptools setuptools-git wheel (in docs/Makefile.*) is unpinned

[zeha]

is python itself also pinned?

[Habbie]

is python itself also pinned?

Python itself is from ~Debian~ Alpine, which we consider trusted, and we do create an artifact with version information from ~dpkg~.

Actually now I wonder if we do that logging for the sdist stage. Let's keep digging :-)