optional=true not respected for local `file` dependency, but works with local `path` dependency

[ellisonch]

Description

Here's a really small reproducible test:

First, grab a .whl to depend on. For simplicity, I'm using six because it has no dependencies:

$ wget https://www.piwheels.org/simple/six/six-1.17.0-py2.py3-none-any.whl#sha256=cb514bebb7be30e172d571b0e5035990437dd202f3c9f6e911d9406254151b92

Now, make a pyproject.toml like this:

[tool.poetry]
name = "test-optional"
version = "0.0.1"

[tool.poetry.dependencies]
python = "^3.9"
six = {optional = true, file = "six-1.17.0-py2.py3-none-any.whl"}

[tool.poetry.extras]
myextra = ["six"]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Now generate the lock file:

$ ~/.local/bin/poetry lock
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Note there was no warning or error. Now look at the lock file and see that six is not considered optional:

$ cat poetry.lock
# This file is automatically @generated by Poetry 2.1.3 and should not be changed by hand.

[[package]]
name = "six"
version = "1.17.0"
description = "Python 2 and 3 compatibility utilities"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*"
groups = ["main"]
markers = "extra == \"myextra\""
files = [
    {file = "six-1.17.0-py2.py3-none-any.whl", hash = "sha256:cb514bebb7be30e172d571b0e5035990437dd202f3c9f6e911d9406254151b92"},
]

[package.source]
type = "file"
url = "six-1.17.0-py2.py3-none-any.whl"

[extras]
myextra = ["six"]

[metadata]
lock-version = "2.1"
python-versions = "^3.9"
content-hash = "3453e3714689a598766b5adad5524b8718b1cc55dc17b28c63922df187265330"

Note that it contains optional = false, which is an error.

Workarounds

If the pyproject.toml file uses path instead of file, everything works:

[tool.poetry]
name = "test-optional"
version = "0.0.1"

[tool.poetry.dependencies]
python = "^3.9"
six = {optional = true, path = "six-1.17.0-py2.py3-none-any.whl"}

[tool.poetry.extras]
myextra = ["six"]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

$ ~/.local/bin/poetry lock
Resolving dependencies... (0.1s)

Writing lock file

$ cat poetry.lock 
# This file is automatically @generated by Poetry 2.1.3 and should not be changed by hand.

[[package]]
name = "six"
version = "1.17.0"
description = "Python 2 and 3 compatibility utilities"
optional = true
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*"
groups = ["main"]
markers = "extra == \"myextra\""
files = [
    {file = "six-1.17.0-py2.py3-none-any.whl", hash = "sha256:cb514bebb7be30e172d571b0e5035990437dd202f3c9f6e911d9406254151b92"},
]

[package.source]
type = "file"
url = "six-1.17.0-py2.py3-none-any.whl"

[extras]
myextra = ["six"]

[metadata]
lock-version = "2.1"
python-versions = "^3.9"
content-hash = "a7c28ec18952f6253307197c341fbf63ce14c757acdc3d1481b70335e1c1ff06"

Poetry Installation Method

install.python-poetry.org

Operating System

Ubuntu

Poetry Version

Poetry (version 2.1.3)

Poetry Configuration

cache-dir = "/home/chucky/.cache/pypoetry"
data-dir = "/home/chucky/.local/share/pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = true
keyring.enabled = true
python.installation-dir = "{data-dir}/python"  # /home/chucky/.local/share/pypoetry/python
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/chucky/.cache/pypoetry/virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false

Python Sysconfig

Platform: "linux-x86_64" Python version: "3.12"

Example pyproject.toml

[tool.poetry]
name = "test-optional"
version = "0.0.1"

[tool.poetry.dependencies]
python = "^3.9"
six = {optional = true, file = "six-1.17.0-py2.py3-none-any.whl"}

[tool.poetry.extras]
myextra = ["six"]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

poetry-runtime.log

$ ~/.local/bin/poetry lock -vvv
...
Resolving dependencies...
 1: fact: test-optional is 0.0.1
 1: derived: test-optional
 1: fact: test-optional depends on six (*)
 1: selecting test-optional (0.0.1)
 1: derived: six (*) @ file:///home/chucky/poetry-bug/six-1.17.0-py2.py3-none-any.whl
 1: selecting six (1.17.0 /home/chucky/poetry-bug/six-1.17.0-py2.py3-none-any.whl)
 1: Version solving took 0.003 seconds.
 1: Tried 1 solutions.

[dimbleby]

you should prefer PEP621 format these days, not tool.poetry.

[project.optional-dependencies]
myextra = [
    "six@file:///home/whatever/path/six-1.17.0-py2.py3-none-any.whl",
]

optional is anyway a mistake in the lock file, and should eventually be removed altogether.

[ellisonch]

@dimbleby

you should prefer PEP621 format these days, not tool.poetry.

I can't use PEP621 for relative path dependencies.

optional is anyway a mistake in the lock file, and should eventually be removed altogether.

This isn't the lock file, it's the pyproject.toml file.

[dimbleby]

I can't use PEP621 for relative path dependencies.

so don't use relative path dependencies

This isn't the lock file, it's the pyproject.toml file.

huh? your whole bug report is about optional in the lock file.

[ellisonch]

I can't use PEP621 for relative path dependencies.

so don't use relative path dependencies

I'm confused. pyproject.toml is shared between our developers, gets checked into version control, and eventually the packages it describes get installed on the cloud. A dependency on a .whl or .zip must be specified relative to the project itself, if it's to be used on more than one machine. Moreover, this is a feature poetry supports. Why are you pushing back on this? What is the alternative?

This isn't the lock file, it's the pyproject.toml file.

huh? your whole bug report is about optional in the lock file.

Ok, maybe I misunderstood your comment. Of course, the problem is that optional is wrong in the lock file, but I'm specifying it's optional in the .toml file. And, things do not work right if optional is set to false in the lock file, as downstream that package is then required to be installed, even though it's meant to be an extra.

Maybe I didn't understand any of your comments.

[dimbleby]

if you are relying on features that are supported only by tool.poetry that should give you pause for thought. You are likely better off doing something more conventional: eg upload the wheel somewhere and use a url dependency, or set up your own package repository.

I am saying that the very existence of optional as a property in the lockfile is a mistake which should eventually be undone.

[ellisonch]

if you are relying on features that are supported only by tool.poetry that should give you pause for thought. You are likely better off doing something more conventional: eg upload the wheel somewhere and use a url dependency, or set up your own package repository.

I am saying that the very existence of optional as a property in the lockfile is a mistake which should eventually be undone.

I see.

Well, let me know if any more information is needed about this bug.

[radoering]

tool.poetry.extras is deprecated so that bugs concerning optional are quite low priority.

In my opinion, relative path dependencies are only for development or projects which are not built but deployed via Poetry. As soon as you are building a wheel I would not expect relative path dependencies to work.

The following should use a relative path dependency when locking and installing via Poetry and add a name/version dependency to the metadata of the built wheel:

[project]
name = "test-optional"
version = "0.0.1"
requires-python = ">=3.9"

[project.optional-dependencies]
myextra = [ "six>=1.17" ]

[tool.poetry.dependencies]
six = {file = "six-1.17.0-py2.py3-none-any.whl"}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"