Add mirrors for pip_install_dependencies

[kgreenek]

🚀 feature request

Relevant Rules

pip_install_dependencies macro from @rules_python//python/pip_install/repositories.bzl

Description

In our CI, we frequently see download errors caused by failing to download dependencies from rules_python. For example, the following bazel error:

Error in download_and_extract: java.io.IOException: Error downloading [https://files.pythonhosted.org/packages/76/0a/b6c5f311e32aeb3b406e03c079ade51e905ea630fc19d1262a46249c1c86/click-8.0.1-py3-none-any.whl] to /tmp/bazel/c75bfb83ea925bcd720c63452511dc13/external/pypi__click/temp13170534750999975287/click-8.0.1-py3-none-any.whl.zip: connect timed out

Our solution is to fork the @rules_python//python/pip_install/repositories.bzl file in our repo, and add additional mirror URL's in case files.pythonhosted.org is failing. This works great, but it is a bit painful to maintain, since we have to update it to keep it in sync whenever updating rules_python.

Describe the solution you'd like

Add additional mirrors for external dependencies to make the downloads more reliable.

Another idea would be to add the ability to specify our own mirrors to the call to pip_parse without forking the repositories.bzl file.

Describe alternatives you've considered

See above.

[driftregion]

@kgreenek , is it possible to use --distdir as a workaround in your situation? I'm using this to eliminate network requests for dependencies in my projects including those listed in @rules_python//python/pip_install/repositories.bzl

https://docs.bazel.build/versions/main/external.html#offline-builds

http_archive (used by pip_install_dependencies) can be instructed to search locally before making a network call. (details: https://github.com/bazelbuild/bazel/issues/9265#issuecomment-525591539)

I realize this isn't exactly what you're asking for, but it might offer some improvement over the current situation.

[kgreenek]

That looks like a helpful suggestion! I will look into using this.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"