pipenv icon indicating copy to clipboard operation
pipenv copied to clipboard

Published 20 hours ago • verified publisher icon pypa

Pipfile with private index AND optional dependency doesn't work

Open scastria opened this issue 1 year ago • 8 comments

I have been using pipenv 2022.9.21 for a long time and decided to try the latest and greatest. The latest 2024.0.1 appears broken when it comes to locating packages in my private AWS CodeArtifact repo.

Issue description

pipenv install fails with this Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-123456789.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
"my_utils[markdown]" = {version="==0.1.391", index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"

I thought it might be a failure in resolving the env variable referenced in the alternate source index called data_engineering so I hardcoded it in the Pipfile. No change.

Expected result

my_utils package is downloaded successfully

Actual result

Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
⠏ Locking packages...False
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement my_utils==0.1.391 (from versions: none)
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/resolver.py", line 645, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/resolver.py", line 612, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/resolver.py", line 592, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 932, in resolve_deps
[ResolutionFailure]:       results, hashes, internal_resolver = actually_resolve_deps(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 700, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 457, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv run pip install <requirement_name> to bypass this mechanism, then run $ pipenv graph to inspect the versions actually installed in the virtualenv.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for my_utils==0.1.391

Traceback (most recent call last):
  File "/opt/homebrew/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/cli/command.py", line 209, in install
    do_install(
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/install.py", line 93, in do_install
    do_init(
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/install.py", line 653, in do_init
    do_lock(
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/lock.py", line 66, in do_lock
    venv_resolve_deps(
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 873, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 737, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate

pipenv install

$ pipenv --support

Pipenv version: '2024.0.1'

Pipenv location: '/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv'

Python location: '/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/bin/python'

OS Name: 'posix'

User pip version: '24.0'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.12.4',
 'os_name': 'posix',
 'platform_machine': 'arm64',
 'platform_python_implementation': 'CPython',
 'platform_release': '23.5.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 23.5.0: Wed May  1 20:12:58 PDT '
                     '2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000',
 'python_full_version': '3.12.4',
 'python_version': '3.12',
 'sys_platform': 'darwin'}

System environment variables:

  • MANPATH
  • TERM_PROGRAM
  • SHELL
  • TERM
  • HISTSIZE
  • HOMEBREW_REPOSITORY
  • TMPDIR
  • GOBIN
  • TERM_PROGRAM_VERSION
  • ADOPS_PAT
  • OLDPWD
  • TERM_SESSION_ID
  • TF_VAR_postman_localhost_api_key
  • FORGE_EMAIL
  • USER
  • LDAP_USERNAME
  • CODEARTIFACT_AUTH_TOKEN
  • SSH_AUTH_SOCK
  • PATH
  • __CFBundleIdentifier
  • PWD
  • JAVA_HOME
  • LANG
  • LDAP_PASSWORD
  • FORGE_API_TOKEN
  • XPC_FLAGS
  • PS1
  • ENVIRONMENT
  • XPC_SERVICE_NAME
  • SHLVL
  • HOME
  • SHELL_SESSION_HISTORY
  • HOMEBREW_PREFIX
  • LOGNAME
  • LOCAL_MYSQL_ROOT_PASSWORD
  • GOPATH
  • INFOPATH
  • HOMEBREW_CELLAR
  • BITBUCKET_PAT
  • HISTTIMEFORMAT
  • _
  • __CF_USER_TEXT_ENCODING
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /opt/homebrew/bin:/opt/homebrew/sbin:/Users/shawncastrianni/.go/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:~/.dotnet/tools:/usr/local/opt/liquibase
  • SHELL: /bin/bash
  • LANG: en_US.UTF-8
  • PWD: /Users/shawncastrianni/GIT/data_api_one

Contents of Pipfile ('/Users/shawncastrianni/GIT/data_api_one/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-123456789.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
"my_utils[markdown]" = {version="==0.1.391", index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"

scastria avatar Jul 05 '24 03:07 scastria

I also tried verify_ssl = false, but no change

scastria avatar Jul 05 '24 03:07 scastria

I discovered a BIG clue to this. It seems to be related to the optional dependency of the my_utils package called markdown. If I remove mention of the optional dependency like this:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-123456789.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
my_utils = {version="==0.1.391", index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"

then the pipenv install command worked. It seems the optional dependency causes a problem. The optional dependency IS VALID as pipenv 2022.9.21 works just fine.

scastria avatar Jul 05 '24 04:07 scastria

Another test is that I removed the my_utils dependency completely from the Pipfile and instead specified ONLY "redis[hiredis]" = "==4.6.0" and that worked too. So optional dependencies work against PyPI, but they do NOT work against private indicies.

scastria avatar Jul 05 '24 04:07 scastria

I wasn't aware of the extras keyword in Pipfile. If I change the Pipfile to use extras, it DOES work. So I guess now the question is should the extras syntax be required or can the other syntax without extras be used?

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-123456789.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
my_utils = {version="==0.1.391", extras=["markdown"], index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"

scastria avatar Jul 05 '24 15:07 scastria

Currently the extras key should be defined, I believe when you run pipenv install "my_utils[markdown]" it should be generating the correct Pipfile entry -- its not clear to me from your report if your initial entry was self input or generated from the install command? Current expectations is the key is just the package name (without extras).

matteius avatar Aug 03 '24 16:08 matteius

Let me try to clarify:

With pipenv 2022.9.21:

  • Pipfile contains {redis = {version="==4.6.0", extras=["hiredis"]} and pipenv install WORKS
  • Pipfile contains "redis[hiredis]" = "==4.6.0" and pipenv install WORKS
  • Pipfile contains my private package my_utils = {version="0.1.391", extras=["markdown"], index="my_private_index"} and pipenv install WORKS
  • Pipfile contains my private package "my_utils[markdown]" = {version="0.1.391", index="my_private_index"} and pipenv install WORKS

With pipenv 2024.0.1:

  • Pipfile contains {redis = {version="==4.6.0", extras=["hiredis"]} and pipenv install WORKS
  • Pipfile contains "redis[hiredis]" = "==4.6.0" and pipenv install WORKS
  • Pipfile contains my private package my_utils = {version="0.1.391", extras=["markdown"], index="my_private_index"} and pipenv install WORKS
  • Pipfile contains my private package "my_utils[markdown]" = {version="0.1.391", index="my_private_index"} and pipenv install DOES NOT WORK

scastria avatar Aug 05 '24 14:08 scastria

2024.* broke something. All my installs fail with it trying to source setuptools from a private repo I have defined as a source. Stepping down to latest 2023.* fixes it.

× pip subprocess to install build dependencies did not run successfully. [pipenv.exceptions.InstallError]: │ exit code: 1 [pipenv.exceptions.InstallError]: ╰─> [3 lines of output] [pipenv.exceptions.InstallError]: Looking in indexes: <REDACTED> [pipenv.exceptions.InstallError]: ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0 (from versions: none) [pipenv.exceptions.InstallError]: ERROR: No matching distribution found for setuptools>=40.8.0 [pipenv.exceptions.InstallError]: [end of output]

elanou avatar Aug 21 '24 22:08 elanou

Could this issue be checked against this branch? https://github.com/pypa/pipenv/pull/6276

matteius avatar Oct 19 '24 22:10 matteius