Python templates: Cannot install bindings to prebuilt providers when behind mirror or proxy of pypi.org

[tpanza]

Expected Behavior

Python bindings for prebuilt providers can be installed from non-standard locations.

For example, if you are in a corporate environment and cannot access pypi.org directly and must go through a proxy/mirror of it, such as JFrog Artifactory or Sonatype Nexus, then users would typically write a ~/.config/pip/pip.conf or set PIP_INDEX_URL environment variable. Users may also want to set PIPENV_PYPI_MIRROR environment variable.

Once these custom config files / env vars are set, then Python packages for bindings for prebuilt providers, such as cdktf-cdktf-provider-docker should use the package manager's custom configuration. It should not attempt to contact pypi.org.

Actual Behavior

With both python and python-pip Python templates, the Python bindings for prebuilt providers are attempted to be downloaded from pypi.org, even though PIP_INDEX_URL and PIPENV_PYPI_MIRROR are set to point to a custom proxy.

Steps to Reproduce

1. Set PIP_INDEX_URL and PIPENV_PYPI_MIRROR env vars to point to an internal mirror of pypi.org. E.g., export PIP_INDEX_URL =https://nexus.example.com/repository/pypi/simple
2. Attempt to init a project with a Python template. E.g., cdktf init --template=python-pip --providers=kreuzwerker/docker --local

Versions

[2024-02-17T02:47:56.034] [ERROR] default - Creating a virtualenv for this project...

Creating a virtualenv for this project... [2024-02-17T02:47:56.054] [ERROR] default - Pipfile: /home/ec2-user/Pipfile

Pipfile: /home/ec2-user/Pipfile [2024-02-17T02:47:56.074] [ERROR] default - Using /usr/bin/python3 (3.9.16) to create virtualenv...

Using /usr/bin/python3 (3.9.16) to create virtualenv... [2024-02-17T02:47:56.513] [ERROR] default - created virtual environment CPython3.9.16.final.0-64 in 207ms creator CPython3Posix(dest=/home/ec2-user/.local/share/virtualenvs/ec2-user-zzkNbF-x, clear=False, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ec2-user/.local/share/virtualenv) added seed packages: pip==23.3.1, setuptools==69.0.2, wheel==0.42.0 activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!

created virtual environment CPython3.9.16.final.0-64 in 207ms creator CPython3Posix(dest=/home/ec2-user/.local/share/virtualenvs/ec2-user-zzkNbF-x, clear=False, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ec2-user/.local/share/virtualenv) added seed packages: pip==23.3.1, setuptools==69.0.2, wheel==0.42.0 activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment! [2024-02-17T02:47:56.552] [ERROR] default - Virtualenv location: /home/ec2-user/.local/share/virtualenvs/ec2-user-zzkNbF-x

Virtualenv location: /home/ec2-user/.local/share/virtualenvs/ec2-user-zzkNbF-x [2024-02-17T02:47:57.385] [ERROR] default - WARNING: Package(s) not found: cdktf

WARNING: Package(s) not found: cdktf [2024-02-17T02:48:02.834] [ERROR] default - WARNING: Package(s) not found: cdktf

WARNING: Package(s) not found: cdktf [2024-02-17T02:48:02.842] [ERROR] default - WARNING: Package(s) not found: jsii

WARNING: Package(s) not found: jsii [2024-02-17T02:48:02.856] [ERROR] default - WARNING: Package(s) not found: constructs

WARNING: Package(s) not found: constructs cdktf debug language: python cdktf-cli: 0.20.3 node: v18.17.1 cdktf: 0.20.3 constructs: 10.3.0 jsii: 1.93.0 terraform: 1.7.3 arch: x64 os: linux 6.1.49-70.116.amzn2023.x86_64 python: Python 3.9.16 pip: pip 23.3.2 from /home/ec2-user/.local/lib/python3.9/site-packages/pip (python 3.9) pipenv: pipenv, version 2023.12.0 providers

Providers

[2024-02-17T02:49:16.801] [ERROR] default - WARNING: Package(s) not found: cdktf

WARNING: Package(s) not found: cdktf ┌┐ ││ └┘

Gist

https://gist.github.com/tpanza/7160a0b21a5fb48799225757c60067d1

Possible Solutions

No response

Workarounds

No response

Anything Else?

No response

References

No response

Help Wanted

• [ ] I'm interested in contributing a fix myself

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[tpanza]

This line might be the issue:

https://github.com/hashicorp/terraform-cdk/blob/main/packages/%40cdktf/cli-core/src/lib/dependencies/package-manager.ts#L343

Looks like it is always hard-coded to use pypi.org.

But there is another complication. AFAIK, on both Artifactory and Nexus, a pypi-typed repository that proxies to pypi.org will not necessarily expose the PyPI's JSON API endpoints.

All we get is the index itself. E.g., https://artifactory.example.com/artifactory/api/pypi/pypi-releases/simple and https://nexus.example.com/repository/pypi/simple

Furthermore, pip search is deprecated and will not work on proxied repositories. Ideally, this would be resolved by just calling pip (which would then pick up whatever custom config exists).

At the moment, however, it appears that this piece of cdktf is architected to make low-level GET requests to the PyPI JSON API endpoints.

[tpanza]

Maybe something like pip install --dry-run ${packageName}==${packageVersion} --report - ?

[tpanza]

Any thoughts on this? Would you accept a PR that solves this?

[DanielMSchmidt]

Sorry for the late reply, we would accept a PR that solves this 👍