Using environment variables with `--extra-index-url` (and `--index-url`)

[ericbn]

This is similar to #966, but in a context where pip-tools can actually keep the variables unexpanded.

Same should apply to the --index-url parameter too.

Environment Versions

1. OS Type: Darwin macbookpro.local 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:07:06 PST 2021; root:xnu-7195.81.3~1/RELEASE_X86_64 x86_64
2. Python version: Python 3.8.8
3. pip version: pip 21.0.1
4. pip-tools version: pip-compile, version 6.0.1

Steps to replicate

1. pip-compile --extra-index-url 'https://${USERNAME}:${PASSWORD}@myrepo/simple/'
2. Open the generated requirements.txt file.

Expected result

Step 1 should evaluate the environment variables instead of prompting for the credentials. It should also accept 'https://$USERNAME:$PASSWORD@myrepo/simple/' without the curly braces. The single quotes are important here, because they prevent the shell from evaluating the environment variables itself.

The generated requirements.txt file should contain:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --extra-index-url='https://${USERNAME}:${PASSWORD}@myrepo/simple/'
#

Actual result

In step 1, the command asks for:

User for myrepo:
Password:

(I had the impression this was working on a previous version, meaning it was not asking for the credentials but instead evaluating the environment variables, or maybe I missed something when I was working on this before)

The generated requirements.txt file contains:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --extra-index-url='https://%24%7BUSERNAME%7D:****@myrepo/simple/'
#

It's escaping the ${} and hiding the password variable.

The generated requirements.txt also contains:

--extra-index-url https://${USERNAME}:${PASSWORD}@myrepo/simple/

which is what we want, so that part is working great! Pip will work with that!

[rundeks]

+1

How are people working around this currently?

[arwedus]

@rundeks : Manually patching the requirements.txt for "--extra-index-url" ☹️

[markliederbach]

We compile without index URLs, then pass in the URL on sync.

python3 -m piptools compile --no-emit-index-url requirements.in --output-file requirements.txt

python3 -m piptools sync --extra-index-url https://${ARTIFACTORY_USERNAME}:${ARTIFACTORY_API_KEY}@foo.com/pypi-local/simple requirements.txt

[threewordphrase]

I am getting around this by using an atrocious sed command on the generated file.

Well, I tried to. eventually ended up just using --no-emit-index-url and relying on setting the index in my Dockerfile.

Would be nice if it worked :)