Read index urls from the requirements files and instead have a flag

[aignas]

Summary from https://github.com/bazel-contrib/rules_python/discussions/2564

pip.parse(
    experimental_bazel_downloader = True,
    experimental_index_url_overrides = {"package": "https//baz.com/simple", ...},
    requirements_lock = "requirements.txt",
    ...

If requirements.txt has:

--index_url https://foo/simple
--extra-index-url https://bar/simple
--extra-index-url https://foobar/simple

package==1.2.3 \
    --hash=sha256:abcd1234

The experimental_bazel_downloader = True API would be functionally equivalent to:

pip.parse(
    experimental_index_url = "https://foo/simple",
    experimental_extra_index_urls = ["https://bar/simple", "https://foobar/simple"]
    experimental_index_url_overrides = {"package": "https//baz.com/simple", ...},  # note no "oauth2accesstoken"
    requirements_lock = "requirements.txt",
    ...

The net result of such a change is that the requirements.in file is now the canonical source for index URLs no matter how the lock file is made:

flowchart TD
    A[req.in] --> B[pip compile<br>uv pip compile]
    A --> C[compile_pip_requirements]
    B --> D[req.lock]
    C --> D
    D --> |effectively sets| E[pip.parse.experimental_index_url<br>pip.parse.experimental_extra_index_urls]

Proposal summary:

1. Remove experimental_index_url, experimental_extra_index_urls
2. Add boolean experimental_bazel_downloader (or similar wording)
3. Have pip.parse pull index URLs from the requirements lock file. If not present, default to public PyPI.

I think keeping experimental_index_url_overrides is prudent.

Alternative:

1. Keep experimental_index_url, experimental_extra_index_urls but support special options like DEFER or similar wording that causes pip.parse to pull index URLs from the requirements lock file.

Originally posted by @dougthor42 in https://github.com/bazel-contrib/rules_python/discussions/2564#discussioncomment-13331380

[martis42]

If a repository is used in different closed off environments, it is possible that each environment uses a different pypi index address (aka the pypi mirror of this environment). Right now this is trivial to model due to

pip-parse(
    hub_name = "some_pip_hub",
    envsubst = ["PIP_INDEX_URL"],
    experimental_index_url = "${PIP_INDEX_URL:-https://pypi.org/simple}",
)

which allows easily changing which pypi index is used. It would be troublesome in such a case, if the pypi index is hard coded via the lock file.

Or should such use cases be handled in the future via --downlaoder_config? Assuming this is the desired solution, does this work for sdist packages? To my understanding for packages not available as wheel experimental_index_url falls back to python -m pip --isolated wheel .... pip would not know about --downlaoder_config.