poetry icon indicating copy to clipboard operation
poetry copied to clipboard
verified publisher icon python-poetry

Optional dependencies for git dependencies aren't resolved with `poetry lock`

Open amitani opened this issue 11 months ago • 0 comments

Description

When optional dependency for a package as a git dependency is updated in pyproject.toml, running poetry.lock fails to add it to install.

Steps to reproduce (checked with 2.1.2 on Mac)

  1. Create a new poetry project with poetry new
  2. Add the non-optional dependency by poetry add sqlalchemy@git+https://github.com/sqlalchemy/sqlalchemy
  3. Edit pyproject.toml, replacing sqlalchemy in dependency to sqlalchemy[postgresql]
  4. Run poetry lock and poetry show psycopg2 is expected to be added, but it doesn't get added.
  5. Run poetry add sqlalchemy[postgresql]@git+https://github.com/sqlalchemy/sqlalchemy This doesn't change pyproject.toml, but updates poetry.lock and poetry show now includes pyscopg2.

sqlalchemy and psycopg2 is just an example. psycopg2 is an optional dependency included in sqlalchemy[postgresql].

This behavior is specific to git dependencies. With pip dependency,

  1. Create a new poetry project with poetry new
  2. Add the non-optional dependency by poetry add sqlalchemy
  3. Edit pyproject.toml, replacing sqlalchemy in dependency to sqlalchemy[postgres]
  4. Run poetry lock and poetry show psycopg2 is included as expected.

Workarounds

Make sure to use poetry add when updating optional dependencies for git dependencies.

Poetry Installation Method

other

Operating System

MacOS (poetry installed with uv tool install

Poetry Version

Poetry (version 2.1.2)

Poetry Configuration

cache-dir = "/Users/akinorimitani/Library/Caches/pypoetry"
data-dir = "/Users/akinorimitani/Library/Application Support/pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = true
keyring.enabled = true
python.installation-dir = "{data-dir}/python"  # /Users/akinorimitani/Library/Application Support/pypoetry/python
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/akinorimitani/Library/Caches/pypoetry/virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false

Python Sysconfig

sysconfig.log 
Paste the output of 'python -m sysconfig', over this line.

Example pyproject.toml

[project]
name = "poetry-extra-git"
version = "0.1.0"
description = ""
authors = [
    {name = "test",email = "test"}
]
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
    "sqlalchemy[postgresql] @ git+https://github.com/sqlalchemy/sqlalchemy"
]

[tool.poetry]
packages = [{include = "poetry_extra_git", from = "src"}]


[build-system]
requires = ["poetry-core>=2.0.0,<3.0.0"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

poetry-runtime.log 
$ poetry -vvv lock
Loading configuration file /Users/akinorimitani/Library/Application Support/pypoetry/config.toml
Using virtualenv: /Users/akinorimitani/Projects/poetry-extra-git/.venv
Checking keyring availability: Checking if keyring is available
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
Using keyring backend 'macOS Keyring'
Available
Resolving dependencies...
 1: fact: poetry-extra-git is 0.1.0
 1: derived: poetry-extra-git
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3:urllib3.connectionpool] https://github.com:443 "GET /sqlalchemy/sqlalchemy/info/refs?service=git-upload-pack HTTP/1.1" 200 None
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (2): github.com:443
[urllib3:urllib3.connectionpool] https://github.com:443 "POST /sqlalchemy/sqlalchemy/git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/sqlalchemy/sqlalchemy at 'HEAD' to /Users/akinorimitani/Projects/poetry-extra-git/.venv/src/sqlalchemy
 1: fact: poetry-extra-git depends on sqlalchemy[postgresql] (2.1.0b1.dev0)
 1: selecting poetry-extra-git (0.1.0)
 1: derived: sqlalchemy[postgresql] (2.1.0b1.dev0) @ git+https://github.com/sqlalchemy/sqlalchemy
 1: fact: sqlalchemy[postgresql] (2.1.0b1.dev0) depends on sqlalchemy (2.1.0b1.dev0)
 1: fact: sqlalchemy[postgresql] (2.1.0b1.dev0) depends on typing-extensions (>=4.6.0)
 1: selecting sqlalchemy[postgresql] (2.1.0b1.dev0 4512255)
 1: derived: typing-extensions (>=4.6.0)
 1: derived: sqlalchemy (2.1.0b1.dev0) @ git+https://github.com/sqlalchemy/sqlalchemy@HEAD
 1: fact: sqlalchemy (2.1.0b1.dev0) depends on typing-extensions (>=4.6.0)
 1: selecting sqlalchemy (2.1.0b1.dev0 4512255)
 1: selecting typing-extensions (4.13.0)
 1: Version solving took 1.316 seconds.
 1: Tried 1 solutions.

Writing lock file

$ poetry -vvv add sqlalchemy[postgresql]@git+https://github.com/sqlalchemy/sqlalchemy
Loading configuration file /Users/akinorimitani/Library/Application Support/pypoetry/config.toml
Using virtualenv: /Users/akinorimitani/Projects/poetry-extra-git/.venv
Checking keyring availability: Checking if keyring is available
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
Using keyring backend 'macOS Keyring'
Available

Updating dependencies
Resolving dependencies...
 1: fact: poetry-extra-git is 0.1.0
 1: derived: poetry-extra-git
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3:urllib3.connectionpool] https://github.com:443 "GET /sqlalchemy/sqlalchemy/info/refs?service=git-upload-pack HTTP/1.1" 200 None
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (2): github.com:443
[urllib3:urllib3.connectionpool] https://github.com:443 "POST /sqlalchemy/sqlalchemy/git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/sqlalchemy/sqlalchemy at 'HEAD' to /Users/akinorimitani/Projects/poetry-extra-git/.venv/src/sqlalchemy
 1: fact: poetry-extra-git depends on sqlalchemy[postgresql] (2.1.0b1.dev0)
 1: selecting poetry-extra-git (0.1.0)
 1: derived: sqlalchemy[postgresql] (2.1.0b1.dev0) @ git+https://github.com/sqlalchemy/sqlalchemy
 1: fact: sqlalchemy[postgresql] (2.1.0b1.dev0) depends on sqlalchemy (2.1.0b1.dev0)
 1: fact: sqlalchemy[postgresql] (2.1.0b1.dev0) depends on typing-extensions (>=4.6.0)
 1: fact: sqlalchemy[postgresql] (2.1.0b1.dev0) depends on psycopg2 (>=2.7)
 1: selecting sqlalchemy[postgresql] (2.1.0b1.dev0 4512255)
 1: derived: psycopg2 (>=2.7)
 1: derived: typing-extensions (>=4.6.0)
 1: derived: sqlalchemy (2.1.0b1.dev0) @ git+https://github.com/sqlalchemy/sqlalchemy@HEAD
Creating new session for pypi.org
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3:urllib3.connectionpool] https://pypi.org:443 "GET /simple/psycopg2/ HTTP/1.1" 304 0
[filelock:filelock] Attempting to acquire lock 4409345648 on /Users/akinorimitani/Library/Caches/pypoetry/cache/repositories/PyPI/_http/d/e/1/c/9/de1c96167763d292fd2ccc8424b2ca35b05bdee33ed2b2d2da5706bb.lock
[filelock:filelock] Lock 4409345648 acquired on /Users/akinorimitani/Library/Caches/pypoetry/cache/repositories/PyPI/_http/d/e/1/c/9/de1c96167763d292fd2ccc8424b2ca35b05bdee33ed2b2d2da5706bb.lock
[filelock:filelock] Attempting to release lock 4409345648 on /Users/akinorimitani/Library/Caches/pypoetry/cache/repositories/PyPI/_http/d/e/1/c/9/de1c96167763d292fd2ccc8424b2ca35b05bdee33ed2b2d2da5706bb.lock
[filelock:filelock] Lock 4409345648 released on /Users/akinorimitani/Library/Caches/pypoetry/cache/repositories/PyPI/_http/d/e/1/c/9/de1c96167763d292fd2ccc8424b2ca35b05bdee33ed2b2d2da5706bb.lock
Source (PyPI): 29 packages found for psycopg2 >=2.7
 1: fact: sqlalchemy (2.1.0b1.dev0) depends on typing-extensions (>=4.6.0)
 1: selecting sqlalchemy (2.1.0b1.dev0 4512255)
 1: selecting typing-extensions (4.13.0)
 1: selecting psycopg2 (2.9.10)
 1: Version solving took 1.508 seconds.
 1: Tried 1 solutions.

Finding the necessary packages for the current system
Source (PyPI): 1 packages found for psycopg2 >=2.7

Package operations: 0 installs, 0 updates, 0 removals, 3 skipped

- Installing psycopg2 (2.9.10): Pending...
- Installing psycopg2 (2.9.10): Skipped for the following reason: Already installed
- Installing typing-extensions (4.13.0): Pending...
- Installing typing-extensions (4.13.0): Skipped for the following reason: Already installed
- Installing sqlalchemy (2.1.0b1.dev0 4512255): Pending...
- Installing sqlalchemy (2.1.0b1.dev0 4512255): Skipped for the following reason: Already installed

Writing lock file

amitani avatar Mar 31 '25 23:03 amitani