Pip extension doesn't support vendoring

[mkruskal-google]

🐞 bug report

Affected Rule

This affects the pip module extension in bzlmod

Is this a regression?

Yes, the old WORKSPACE setup didn't have this issue

Description

We do a two stage build process as a security measure to prevent unexpected network access. The first step runs:

bazel build ... --nobuild --vendor_dir=vendor

In the second step we disable network access and run:

bazel build ... --vendor_dir=vendor

(note: this is also reproducible with --nofetch instead of --vendor_dir)

Despite all of our pip dependencies getting downloaded during the first step, we still see warnings in the second when the extension tries to download from pypi again, such as:

WARNING: Download from https://pypi.org/simple/certifi/ failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Unknown host: pypi.org

This eventually results in a failed build.

🔬 Minimal Reproduction

• Checkout https://github.com/protocolbuffers/protobuf at main
• Run bazel build //python/dist:source_wheel --nobuild --//toolchain:release=true --vendor_dir=vendor
• Disable network access
• Run bazel build //python/dist:source_wheel --//toolchain:release=true --vendor_dir=vendor

🔥 Exception or Error

After multiple failed download warnings we eventually hit a build error:

ERROR: /workspace/_build/out/external/rules_python~/python/private/pypi/simpleapi_download.bzl:139:14: Traceback (most recent call last):
	File "/workspace/_build/out/external/rules_python~/python/private/pypi/extension.bzl", line 873, column 25, in _pip_impl
		mods = parse_modules(module_ctx, enable_pipstar = rp_config.enable_pipstar)
	File "/workspace/_build/out/external/rules_python~/python/private/pypi/extension.bzl", line 723, column 36, in parse_modules
		out = _create_whl_repos(
	File "/workspace/_build/out/external/rules_python~/python/private/pypi/extension.bzl", line 257, column 50, in _create_whl_repos
		requirements_by_platform = parse_requirements(
	File "/workspace/_build/out/external/rules_python~/python/private/pypi/parse_requirements.bzl", line 171, column 36, in parse_requirements
		index_urls = get_index_urls(
	File "/workspace/_build/out/external/rules_python~/python/private/pypi/extension.bzl", line 698, column 79, in lambda
		get_index_urls = lambda ctx, distributions: simpleapi_download(
	File "/workspace/_build/out/external/rules_python~/python/private/pypi/simpleapi_download.bzl", line 139, column 14, in simpleapi_download
		_fail(
Error in fail: 
Failed to download metadata of the following packages from urls:

🌍 Your Environment

Operating System: Linux Output of bazel version: 7 and 8 Rules_python version: 1.6.0 Anything else relevant?

[aignas]

Is this a duplicate of #2937?

[aignas]

With #2949 having been closed, vendoring should work even better than before.