`whl_library.BuildWheelFromSource` uses pip to download dependencies and does not inject credential helper information.

[dougthor42]

🐞 bug report

Affected Rule

pip.parse and the underlying code

Is this a regression?

Not that I can tell.

Description

When using python package index that requires authentication, whl_library.BuildWheelFromSource will pass that index arg to the pip wheel command.

This is a problem when using the Bazel downloader (experimental_index_url et. al.). Bazel can authenticate to the package index via a credential helper header injection and can successfully download the source tarball. However, rules_python then tries to create a wheel from that tarball and uses pip wheel --index-url "${INDEX_NEEDING_AUTH}" .... Thus building the wheel fails because pip can't auth to the private index.

🔬 Minimal Reproduction

Hard to repro if you don't have a private index readily available, but the gist is:

1. Use only package indexes that require authentication
2. Use experimental Bazel downloader experimental_index_url
3. Attempt to install a package that fits both of these requirements. pygraphviz is a good example of such a package.
   1. Does not have a wheel available on the package index
   2. Requires an additional package (such as setuptools) in order to build a wheel.

🔥 Exception or Error

root@b4337bd118dd:/bazel_starter# time bazel build --nobuild --config=local --remote_cache= --bes_backend= //...
INFO: Repository rules_python~~pip~pypi_312_pygraphviz_sdist_8b0b9207 instantiated at:
  <builtin>: in <toplevel>
Repository rule whl_library defined at:
  /root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~/python/private/pypi/whl_library.bzl:469:30: in <toplevel>
INFO: repository @@rules_python~~pip~pypi_312_pygraphviz_sdist_8b0b9207' used the following cache hits instead of downloading the corresponding file.
 * Hash '8b0b9207954012f3b670e53b8f8f448a28d12bdbbcf69249313bd8dbe680152f' for https://REDACTED_1/pygraphviz/pygraphviz-1.12.tar.gz
If the definition of 'repository @@rules_python~~pip~pypi_312_pygraphviz_sdist_8b0b9207' was updated, verify that the hashes were also updated.
ERROR: /root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~/python/private/repo_utils.bzl:79:16: An error occurred during the fetch of repository 'rules_python~~pip~pypi_312_pygraphviz_sdist_8b0b9207':
   Traceback (most recent call last):
        File "/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~/python/private/pypi/whl_library.bzl", line 245, column 40, in _whl_library_impl
                pypi_repo_utils.execute_checked(
        File "/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~/python/private/pypi/pypi_repo_utils.bzl", line 133, column 38, in _execute_checked
                return repo_utils.execute_checked(
        File "/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~/python/private/repo_utils.bzl", line 216, column 29, in _execute_checked
                return _execute_internal(fail_on_error = True, *args, **kwargs)
        File "/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~/python/private/repo_utils.bzl", line 147, column 27, in _execute_internal
                return logger.fail((
        File "/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~/python/private/repo_utils.bzl", line 89, column 39, in lambda
                fail = lambda message_cb: _log(-1, "FAIL", message_cb, fail),
        File "/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~/python/private/repo_utils.bzl", line 79, column 16, in _log
                printer("\nrules_python:{} {}:".format(
Error in fail:
rules_python:whl_library(@@rules_python~~pip~pypi_312_pygraphviz_sdist_8b0b9207) FAIL: repo.execute: whl_library.BuildWheelFromSource(rules_python~~pip~pypi_312_pygraphviz_sdist_8b0b9207, pygraphviz==1.12): end: failure:
  command: /root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~python~python_3_12_2_host/python -m python.private.pypi.whl_installer.wheel_installer --requirement pygraphviz==1.12 --isolated --extra_pip_args "{\"arg\":[\"--index-url\",\"https://oauth2accesstoken@REDACTED_2/simple\",\"--extra-index-url\",\"https://oauth2accesstoken@REDACTED_1/simple\",\"--find-links\",\".\"]}" --pip_data_exclude "{\"arg\":[]}" --environment "{\"arg\":{}}"
  return code: 1
  working dir: <default: /root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~pip~pypi_312_pygraphviz_sdist_8b0b9207>
  timeout: 600
  environment:
PYTHONPATH="/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~pypi__build:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~pypi__click:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~py
pi__colorama:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~pypi__importlib_metadata:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~pypi__installer:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~pypi__more_itertools:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3
aa1b8af741d50/external/rules_python~~internal_deps~pypi__packaging:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~pypi__pep517:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~pypi__pip:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~pypi__pip_tools:/root/.cache/bazel/
_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~pypi__pyproject_hooks:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~pypi__setuptools:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~pypi__tomli:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~intern
al_deps~pypi__wheel:/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~internal_deps~pypi__zipp"
CPPFLAGS="-isystem /root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~python~python_3_12_2_host/include/python3.12"
===== stdout start =====
Looking in indexes: https://****@REDACTED_2/simple, https://****@REDACTED_1/simple                                                                                                                                                                                                                                                                                         Looking in links: .
Processing ./pygraphviz-1.12.tar.gz (from -r /tmp/tmp192c1dbh (line 1))
  File was already downloaded /root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~pip~pypi_312_pygraphviz_sdist_8b0b9207/pygraphviz-1.12.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
===== stdout end =====
===== stderr start =====
WARNING: 401 Error, Credentials not correct for https://REDACTED_2/simple/pygraphviz/
WARNING: 401 Error, Credentials not correct for https://REDACTED_1/simple/pygraphviz/
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Looking in indexes: https://****@REDACTED_2/simple, https://****@REDACTED_1/simple
      Looking in links: .
      WARNING: 401 Error, Credentials not correct for https://REDACTED_2/simple/setuptools/
      WARNING: 401 Error, Credentials not correct for https://REDACTED_1/simple/setuptools/
      ERROR: Could not find a version that satisfies the requirement setuptools>=61.2 (from versions: none)
      ERROR: No matching distribution found for setuptools>=61.2
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 205, in <module>
    main()
  File "/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 190, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~python~python_3_12_2_x86_64-unknown-linux-gnu/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/root/.cache/bazel/_bazel_root/dd66987607986e0bbc3aa1b8af741d50/external/rules_python~~python~python_3_12_2_host/python', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--index-url', 'https://oauth2accesstoken@REDACTED_2/simple', '--extra-index-url', 'https://oauth2accesstoken@REDACTED_1/simple', '--find-links', '.',
 '-r', '/tmp/tmp192c1dbh']' returned non-zero exit status 1.
===== stderr end =====

🌍 Your Environment

Operating System:

gLinux (based on Debian testing)

Output of bazel version:

$ bazel version
Bazelisk version: v1.20.0
Starting local Bazel server and connecting to it...
Build label: 7.4.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Nov 11 21:24:53 2024 (1731360293)
Build timestamp: 1731360293
Build timestamp as int: 1731360293

Rules_python version:

1.1.0

Anything else relevant?

Internal bug: b/399782261

I think I see a couple potential paths forward:

1. support keyring when building wheels
   • At least with some private registries, they support using keyring as an auth provider. This may allow the internal pip wheel command to auth to the registry.
2. support using the Bazel downloader, and thus the credential helper, when building wheels
   • Manually replacing pip wheel's downloading action with a separate Bazel download task might work, but only if people are using the Bazel downloader
3. support passing a secret value (the access token) down to the pip wheel command and injecting that secret into the (extra) index url.
   • As far as I know, all package indexes support auth via URL credential injection a-la https://oauth2accesstoken:${SECRET}@private_index.com/simple. It might be possible to add an attribute to pip.parse that injects that secret.

[aignas]

@dougthor42, I think you had some experiments with adding more deps to the pip parse to include credetial support, but that was somewhat hard to complete.

Related, I think we should focus on #2410 so that we don't have a need to extend the pip dependencies to support credentials.

Potentially related: #2328, where at some point we removed the --no-index-url from the code branch in question.

[dougthor42]

FYI: I'm currently working around this with the following patch, with some things redacted. Hopefully I didn't accidentally remove any lines in the copy-paste-redact or else the patch line numbers will be all mucked up 🙃.

diff --git a/python/private/pypi/whl_library.bzl b/python/private/pypi/whl_library.bzl
index 630dc851..6189946d 100644
--- a/python/private/pypi/whl_library.bzl
+++ b/python/private/pypi/whl_library.bzl
@@ -289,6 +289,38 @@ def _whl_library_impl(rctx):
         if filename.endswith(".whl"):
             whl_path = rctx.path(rctx.attr.filename)
         else:
+            # HACK ALERT
+            # We need to inject a secret into the pypi index url if we are not using any public
+            # package index.
+            # TODO: Remove this hack when rules_python issue is fixed and released.
+            _index_path = "us-python.pkg.dev/REDACTED/REDACTED/simple"
+            _url = "https://oauth2accesstoken@{}".format(_index_path)
+            if _url in extra_pip_args:
+                print("{} is not a wheel and we are using Airlock. Injecting gcloud access token.".format(filename))
+
+                # N.B.: This is a nontrivial time cost of ~1s/call, but we only have ~13 sdist deps.
+                secret = repo_utils.execute_checked_stdout(
+                    rctx,
+                    op = "GetGCloudAuthAccessToken",
+                    arguments = ["gcloud", "auth", "print-access-token"],
+                    log_stdout = False,
+                )
+
+                # extra_pip_args is a list. It's safe enough to assume that the element after
+                # the "--index-url" string is the URL because otherwise the value
+                # would not be attached to the correct CLI arg.
+                # We must remove --index-url from `extra_pip_args` because of variable
+                # precedence: CLI args take precedence over env vars (see
+                # https://pip.pypa.io/en/stable/topics/configuration/#precedence-override-order).
+                extra_pip_args.remove(_url)
+                arg_index = extra_pip_args.index("--index-url")
+                extra_pip_args.remove(extra_pip_args[arg_index])
+
+                prefix = "https://oauth2accesstoken:{}@".format(secret.strip())
+                environment["PIP_INDEX_URL"] = prefix + _index_path
+
+            # END HACK
+
             # It is an sdist and we need to tell PyPI to use a file in this directory
             # and, allow getting build dependencies from PYTHONPATH, which we
             # setup in this repository rule, but still download any necessary
diff --git a/python/private/repo_utils.bzl b/python/private/repo_utils.bzl
index eee56ec8..3c8fa287 100644
--- a/python/private/repo_utils.bzl
+++ b/python/private/repo_utils.bzl
@@ -338,10 +338,21 @@ def _cwd_to_str(mrctx, kwargs):
     return cwd
 
 def _env_to_str(environment):
-    if not environment:
+    # HACK ALERT
+    # TODO: Remove this hack when rules_python issue is fixed and released.
+    _env = dict(environment)  # N.B.: .copy not supported
+    if "PIP_INDEX_URL" in environment.keys():
+        # This could definitely be better so that we log something like
+        # "PIP_INDEX_URL=https://oauthaccesstoken:[email protected]/..."
+        # but it's not worth it right now.
+        _env["PIP_INDEX_URL"] = "<REDACTED>"
+
+    # END HACK
+
+    if not _env:
         env_str = " <default environment>"
     else:
-        env_str = "\n".join(["{}={}".format(k, repr(v)) for k, v in environment.items()])
+        env_str = "\n".join(["{}={}".format(k, repr(v)) for k, v in _env.items()])
         env_str = "\n" + env_str
     return env_str

[rickeylev]

Have you tried a netrc file? https://pip.pypa.io/en/stable/topics/authentication/#netrc-support

pip.parse has a netrc arg: https://rules-python.readthedocs.io/en/latest/api/rules_python/python/extensions/pip.html#pip.parse.netrc

[aignas]

FYI, ~/.netrc is picked up by default.

[dougthor42]

.netrc... that's a name I haven't heard in a long time. So long that I forgot about it!

I did some testing (thanks to @binarystardust for actually doing the testing - I just copied his work 🙃) and it does indeed work. Now I just have to verify with our security team that using .netrc is OK, as it does still store plaintext secrets.

[tkinz27]

Its not quite hermetic, but I get this to work with by adding

    extra_pip_args = ["--keyring-provider=subprocess"],

to our pip.parse method and then ensuring our developers have keyring installed in their PATH. We are using Google Artifact Registry. My whole pip.parse looks like

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    enable_implicit_namespace_pkgs = True,
    experimental_index_url = "https://[email protected]/********/python-virtual/simple/",
    experimental_target_platforms = [
        "host",
        "linux_x86_64",
    ],
    extra_pip_args = ["--keyring-provider=subprocess"],
    hub_name = "python_deps",
    # MUST MATCH python.toolchain above
    python_version = "3.10",
    requirements_by_platform = {
        "//:requirements_linux_x86_64.txt": "linux_x86_64",
        "//:requirements_linux_arm64.txt": "linux_aarch64",
        "//:requirements_macos_x86_64.txt": "osx_x86_64",
        "//:requirements_macos_arm64.txt": "osx_aarch64",
    },
)
use_repo(pip, "python_deps")

[aignas]

In the long term #2410 is the real solution, so just linking the issues together.