pip_parse: lockfile for build dependencies in repository rule

[keith]

Since PEP 518 projects can specify their build system requirements in their pyproject.toml file. Currently this is not included in the pip-compile and therefore can lead to non-reproducible issues. This happened in the wild today because pyyaml depends on cython but didn't pin its major version, so when cython pushed an incompatible version this broke if you were including it in bazel. One of the potential workarounds is to install the compatible cython version in the venv before installing your requirements, which I also don't believe is possible. I believe supporting this would require pip-tools to also support this option, which is potentially implemented by https://github.com/jazzband/pip-tools/pull/1681

[keith]

Another potentially option is multiple requirements files. I believe 1 method that would work is if there was a tooling requirements file that was installed first, and then the normal requirements file which was installed inside the venv used with the first requirements file.

[groodt]

There are a number of conflated issues here. One thing to note is that pip does not support pinning build dependencies. See: https://discuss.python.org/t/no-way-to-pin-build-dependencies/29833

Generally, the pip_parse rule is always going to be non-hermetic since it runs inside repository rules. If your dependency closure includes an sdist, pip will generally build with "build-isolation", where the build dependencies are installed into a temporary virtual environment. This happens on your machine and isn't managed by bazel. Typically, this is what you would normally want. However, sometimes there are missed dependencies specified in a dependency, or, as you've encountered, you wish to prepare the build environment ahead of time. For this, pip provides --no-build-isolation. You are able to provide this to the bazel repository rule, and you are able to prepare a build environment ahead of time on your build host. This isn't managed by bazel however.

I suspect the above is not what you want to hear, but the flexibility and impurity of the full Python ecosystem that is comprised of Turing complete and non-declarative build specs does not fit very well with the pure and declarative / hermetic world of bazel.

Some possible options that you may wish to consider:

• Try rules_pycross which will run builds outside of a repository rule context
• Make use of a private package registry / wheel-house / cheese-shop. If you only make use of pre-built wheels in your dependency closure, then bazel will not invoke the build+unpack behaviour of pip and you'll be able to use --only-binary :all: in your requirements.txt. In this configuration, bazel will simply download and unpack wheels and you will not need to be concerned with unmanaged build dependencies.

[keith]

In the linked thread the solution of using PIP_CONSTRAINT=foo.txt might actually be a good path forward for this. I verified that does work for this case.

[groodt]

Great! Yes, if that works for you, we may just consider that the recommended workaround and eventually close this issue.

[keith]

I'm not sure that's possible to integrate with rules_python today though

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"

[aignas]

Reopening as the feature for including build deps in pip-tools got merged so the definition of done here could be to ensure that it is available in rules_python.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!