Add support for config vars in Pip requirements files

[ericwb]

Unsure if I'm somehow doing something wrong or this is by design. According to Pip's documentation, I can include a private repo in my requirements.txt as a dependency. Since the repo is private and hosted on GitHub, I need to provide a personal access token. Of course, I do not want that token included in source code, so I configure it in Heroku's Config Vars. But it doesn't seem to work for me. However works fine when running locally.

For example, requirements.txt:

aiohttp
cryptography
cachetools
sqlalchemy==1.4.46
git+https://${GITHUB_TOKEN}@github.com/user/repo.git@main

This is the error I get from Heroku. It appears, it does not resolve the GITHUB_TOKEN environmental variable that is defined in the Config Vars.

-----> Installing requirements with pip
       Collecting git+https://****@github.com/user/repo.git@main (from -r requirements.txt (line 9))
         Cloning https://****@github.com/user/repo.git (to revision main) to /tmp/pip-req-build-imw3lwma
         Running command git clone --filter=blob:none --quiet 'https://****@github.com/user/repo.git' /tmp/pip-req-build-imw3lwma
         fatal: could not read Password for 'https://${GITHUB_TOKEN}@github.com': No such device or address
         error: subprocess-exited-with-error

Is there a workaround or something I'm missing? Thanks

https://pip.pypa.io/en/stable/reference/requirements-file-format/#using-environment-variables

[ericwb]

I should also add that the Heroku docs state: "Anything that works with a standard pip requirements file will work as expected on Heroku."

However, following that statement, it gives an example that is unsupported on GitHub. GitHub no longer supports user/password based authentication.

git+https://user:[email protected]/nsa/secret.git

Support for password authentication was removed on August 13, 2021. Please use a personal access token instead.
remote: Please see https://github.blog/2020-12-15-token-authentication-requirements-for-git-operations/ for more information.

https://devcenter.heroku.com/articles/python-pip#git-backed-distributions

[ericwb]

I guess this issue has come up several times in different forms: https://github.com/heroku/heroku-buildpack-python/pull/417 https://github.com/heroku/heroku-buildpack-python/pull/467 https://github.com/heroku/heroku-buildpack-python/pull/639

The only workaround I seem to have is hosting another python package index and using PIP_EXTRA_INDEX_URL. Which is a hacky option for one repo.

[ericwb]

This link seems to provide a good option on how to modify heroku-buildpack-python to utilize and acceptlist and denylist for env vars. Maybe this would be a worthy consideration as a fix?

https://devcenter.heroku.com/articles/buildpack-api#bin-compile-usage

export_env_dir() {
  env_dir=$1
  acceptlist_regex=${2:-''}
  denylist_regex=${3:-'^(PATH|GIT_DIR|CPATH|CPPATH|LD_PRELOAD|LIBRARY_PATH)$'}
  if [ -d "$env_dir" ]; then
    for e in $(ls $env_dir); do
      echo "$e" | grep -E "$acceptlist_regex" | grep -qvE "$denylist_regex" &&
      export "$e=$(cat $env_dir/$e)"
      :
    done
  fi
}

[edmorley]

@ericwb Hi! Thank you for filing this, and sorry for the delay (I was away).

You are correct that env vars are filtered out in this buildpack at present for certain subprocesses, with pip install being one of them.

I'd like to revisit that decision in the new Python Cloud Native Buildpack (that's set to replace this one) - I've filed: https://github.com/heroku/buildpacks-python/issues/52

In the meantime, I'd recommend trying out .netrc file to configure auth - see: https://pip.pypa.io/en/stable/topics/authentication/#netrc-support

Since you are using GitHub for the package location, the .netrc file can be created using: https://github.com/heroku/heroku-buildpack-github-netrc

(For anyone else not using GitHub for the package location, then you will need to either fork the above buildpack, or else manually create a .netrc file using eg https://github.com/heroku/heroku-buildpack-inline)