terraform-cdk
terraform-cdk copied to clipboard
Python templates: Cannot install bindings to prebuilt providers when behind mirror or proxy of pypi.org
Expected Behavior
Python bindings for prebuilt providers can be installed from non-standard locations.
For example, if you are in a corporate environment and cannot access pypi.org directly and must go through a proxy/mirror of it, such as JFrog Artifactory or Sonatype Nexus, then users would typically write a ~/.config/pip/pip.conf
or set PIP_INDEX_URL
environment variable. Users may also want to set PIPENV_PYPI_MIRROR
environment variable.
Once these custom config files / env vars are set, then Python packages for bindings for prebuilt providers, such as cdktf-cdktf-provider-docker
should use the package manager's custom configuration. It should not attempt to contact pypi.org
.
Actual Behavior
With both python
and python-pip
Python templates, the Python bindings for prebuilt providers are attempted to be downloaded from pypi.org
, even though PIP_INDEX_URL
and PIPENV_PYPI_MIRROR
are set to point to a custom proxy.
Steps to Reproduce
- Set
PIP_INDEX_URL
andPIPENV_PYPI_MIRROR
env vars to point to an internal mirror ofpypi.org
. E.g.,export PIP_INDEX_URL =https://nexus.example.com/repository/pypi/simple
- Attempt to init a project with a Python template. E.g.,
cdktf init --template=python-pip --providers=kreuzwerker/docker --local
Versions
[2024-02-17T02:47:56.034] [ERROR] default - Creating a virtualenv for this project...
Creating a virtualenv for this project... [2024-02-17T02:47:56.054] [ERROR] default - Pipfile: /home/ec2-user/Pipfile
Pipfile: /home/ec2-user/Pipfile [2024-02-17T02:47:56.074] [ERROR] default - Using /usr/bin/python3 (3.9.16) to create virtualenv...
Using /usr/bin/python3 (3.9.16) to create virtualenv... [2024-02-17T02:47:56.513] [ERROR] default - created virtual environment CPython3.9.16.final.0-64 in 207ms creator CPython3Posix(dest=/home/ec2-user/.local/share/virtualenvs/ec2-user-zzkNbF-x, clear=False, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ec2-user/.local/share/virtualenv) added seed packages: pip==23.3.1, setuptools==69.0.2, wheel==0.42.0 activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
✔ Successfully created virtual environment!
created virtual environment CPython3.9.16.final.0-64 in 207ms creator CPython3Posix(dest=/home/ec2-user/.local/share/virtualenvs/ec2-user-zzkNbF-x, clear=False, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ec2-user/.local/share/virtualenv) added seed packages: pip==23.3.1, setuptools==69.0.2, wheel==0.42.0 activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
✔ Successfully created virtual environment! [2024-02-17T02:47:56.552] [ERROR] default - Virtualenv location: /home/ec2-user/.local/share/virtualenvs/ec2-user-zzkNbF-x
Virtualenv location: /home/ec2-user/.local/share/virtualenvs/ec2-user-zzkNbF-x [2024-02-17T02:47:57.385] [ERROR] default - WARNING: Package(s) not found: cdktf
WARNING: Package(s) not found: cdktf [2024-02-17T02:48:02.834] [ERROR] default - WARNING: Package(s) not found: cdktf
WARNING: Package(s) not found: cdktf [2024-02-17T02:48:02.842] [ERROR] default - WARNING: Package(s) not found: jsii
WARNING: Package(s) not found: jsii [2024-02-17T02:48:02.856] [ERROR] default - WARNING: Package(s) not found: constructs
WARNING: Package(s) not found: constructs cdktf debug language: python cdktf-cli: 0.20.3 node: v18.17.1 cdktf: 0.20.3 constructs: 10.3.0 jsii: 1.93.0 terraform: 1.7.3 arch: x64 os: linux 6.1.49-70.116.amzn2023.x86_64 python: Python 3.9.16 pip: pip 23.3.2 from /home/ec2-user/.local/lib/python3.9/site-packages/pip (python 3.9) pipenv: pipenv, version 2023.12.0 providers
Providers
[2024-02-17T02:49:16.801] [ERROR] default - WARNING: Package(s) not found: cdktf
WARNING: Package(s) not found: cdktf ┌┐ ││ └┘
Gist
https://gist.github.com/tpanza/7160a0b21a5fb48799225757c60067d1
Possible Solutions
No response
Workarounds
No response
Anything Else?
No response
References
No response
Help Wanted
- [ ] I'm interested in contributing a fix myself
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
This line might be the issue:
https://github.com/hashicorp/terraform-cdk/blob/main/packages/%40cdktf/cli-core/src/lib/dependencies/package-manager.ts#L343
Looks like it is always hard-coded to use pypi.org
.
But there is another complication. AFAIK, on both Artifactory and Nexus, a pypi-typed repository that proxies to pypi.org will not necessarily expose the PyPI's JSON API endpoints.
All we get is the index itself. E.g., https://artifactory.example.com/artifactory/api/pypi/pypi-releases/simple
and https://nexus.example.com/repository/pypi/simple
Furthermore, pip search
is deprecated and will not work on proxied repositories. Ideally, this would be resolved by just calling pip
(which would then pick up whatever custom config exists).
At the moment, however, it appears that this piece of cdktf is architected to make low-level GET
requests to the PyPI JSON API endpoints.
Maybe something like pip install --dry-run ${packageName}==${packageVersion} --report -
?
Any thoughts on this? Would you accept a PR that solves this?
Sorry for the late reply, we would accept a PR that solves this 👍