pytorch and poetry

[yonimedhub]

• [X ] I am on the latest Poetry version.
• [X ] I have searched the issues of this repo and believe that this is not a duplicate.
• [ X] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: docker on linux with cuda
• Poetry version: 1.1.7
• Link of a Gist with the contents of your pyproject.toml file: <below

Issue

pytorch is now pep503 compliant (https://github.com/pytorch/pytorch/issues/25639#issuecomment-861707149) but I still can't add torch and torchvision. I've tried using this:

torch = {url = "https://download.pytorch.org/whl/cu111/torch-1.9.0%2Bcu111-cp38-cp38-linux_x86_64.whl"}
torchvision = {url = "https://download.pytorch.org/whl/cu111/torchvision-0.10.0%2Bcu111-cp38-cp38-linux_x86_64.whl"}

but got this:

Updating dependencies
Resolving dependencies... (326.0s)

  SolverProblemError

  Because torchvision (0.10.0+cu111) depends on torch (1.9.0)
   and  depends on torch (1.9.0+cu111), torchvision is forbidden.
  So, because  depends on torchvision (0.10.0+cu111), version solving failed.

and if I try using

 [[tool.poetry.source]]
 name = "torch_rep"
 url = "https://download.pytorch.org/whl/cu111/"

I end up disabling the pypi and can't download anything else...

[Piroro-hs]

[tool.poetry.dependencies]
torch = { version = "=1.9.0+cu111", source = "pytorch" }
torchvision = { version = "=0.10.0+cu111", source = "pytorch" }

[[tool.poetry.source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu111/"
secondary = true

This works for me with poetry v1.2.0a1 (poetry install is so slow due to #4035, though).

[yonimedhub]

this is my pyproject.toml:

[tool.poetry]
name = ""
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]

[tool.poetry.dependencies]
python = "3.8.5"
numpy = "^1.21.0"
matplotlib = "^3.4.2"
torch = { version = "=1.9.0+cu111", source = "pytorch" }
torchvision = { version = "=0.10.0+cu111", source = "pytorch" }

[[tool.poetry.source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu111/"
secondary = true

[tool.poetry.dev-dependencies]


[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

with this I get the following error (with poetry update):

  SolverProblemError

  Because torchvision (0.10.0+cu111) depends on torch (1.9.0)
   and  depends on torch (=1.9.0+cu111), torchvision is forbidden.
  So, because  depends on torchvision (=0.10.0+cu111), version solving failed.

and if I put both torch and torchvision in comments I still get the following error:

  RepositoryError

  403 Client Error: Forbidden for url: https://download.pytorch.org/whl/cu111/matplotlib/

[renan-r-santos]

This is a duplicate of #2543, #3855, #3306 and some others.

An ugly workaround while this is not fixed is:

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["author <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.9"

[tool.poe.tasks]
## PyTorch with CUDA 11.1. If PyTorch is imported first, importing Tensorflow will detect CUDA + cuDNN bundled with PyTorch
## Run with the command "poe force-cuda11"
## See https://github.com/python-poetry/poetry/issues/2543
force-cuda11 = "pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html"

[tool.poetry.dev-dependencies]
black = "^21.6b0"
poethepoet = "^0.10.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[wangm23456]

@scherzocrk when I "poetry install", the torch will be " • Updating torch (1.9.0+cu111 -> 1.9.0)"

[kikohs]

@wangm23456 I wrote the hack above using the task-runner, I have the same issue, poetry always rolls back to regular torch. It gets me crazy..

[bennylut]

[tool.poetry.dependencies]
torch = { version = "=1.9.0+cu111", source = "pytorch" }
torchvision = { version = "=0.10.0+cu111", source = "pytorch" }

[[tool.poetry.source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cu111/"
secondary = true

This works for me with poetry v1.2.0a1 (poetry install is so slow due to #4035, though).

Just tested it on v1.2.0a2 and with the secondary source poetry install is still very slow and takes more than 18m in my machine.

[david-waterworth]

Also worse than not being able to install torchvision, for the same reason you cannot install any ML package which depends on torch. For example allennlp requires torch>=1.6.0,<1.11.0 and torchvision>=0.8.1,<0.12.0, so the result is:

  Because no versions of allennlp match >2.8.0,<3.0.0
   and allennlp (2.8.0) depends on torchvision (>=0.8.1,<0.12.0), allennlp (>=2.8.0,<3.0.0) requires torchvision (>=0.8.1,<0.12.0).
  Thus, allennlp (>=2.8.0,<3.0.0) requires torch (1.7.0 || 1.7.1 || 1.8.0 || 1.8.1 || 1.9.0 || 1.9.1 || 1.10.0 || 1.10.0+cu102).
  So, because disambiguation depends on both torch (1.9.1+cu111) and allennlp (^2.8.0), version solving failed.

[psinger]

Coming back half a year later it seems there is still no solution for Pytorch related packages in poetry. Will this ever get solved? Otherwise need to move to other solutions.

[mhnatiuk]

yeah, about that. I want to add this functionality, so my team can use poetry. Otherwise, we'll stick to conda, which is not optimal, bc poetry plays so well with CI & CD pipelines, espescially for packaging software

[psinger]

Yeah, we unfortunately had to decide now to move away from poetry to good old pipenv. Let's hope there will be a solution as some point for poetry.

[Kavan72]

@psinger here is the good news for you. I just tested with poetry version 1.2.0b1 for cpu and cuda version of pytorch and both are working fine for me. Here is the my dependencies

[tool.poetry.dependencies]
python = "^3.8"
fastapi = "^0.75.0"
gunicorn = "^20.1.0"
loguru = "^0.6.0"
torch = {url = "https://download.pytorch.org/whl/cpu/torch-1.11.0%2Bcpu-cp38-cp38-linux_x86_64.whl"}
torchaudio = {url = "https://download.pytorch.org/whl/cpu/torchaudio-0.11.0%2Bcpu-cp38-cp38-linux_x86_64.whl"}
torchvision = {url = "https://download.pytorch.org/whl/cpu/torchvision-0.12.0%2Bcpu-cp38-cp38-linux_x86_64.whl"}
transformers = "^4.17.0"

[dan-ringwald]

@Kavan72 thx for the update, it works.

However the dependencies resolving time is absolutely huge (600+ seconds), would it be possible to ignore dependency solving for a package, or hard code the dependencies so that poetry doesn't have to full scan a 1.5Go package ?

[kikohs]

The solution of @Kavan72 is cool but unfortunately still a work around as you need to target a specific python version, and OS which goes against the whole idea of Poetry. My ugly workaround (with pip install) is not better either...

[dan-ringwald]

@kikohs I ended up using poe the poet as a task runner too in the end. To avoid rolling back to cpu torch, you have to install every torch-using lib with pip through the task-runner too (e.g. transformers)

[suneeta-mall]

This issue is blocking "unlocking the poetry" potentials when it comes to leveraging poetry in the stack that uses PyTorch and its ecosystem.

I have tried with following: toml setting

[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/cu113"
secondary = true
default = false

but the default false is not recognized for some reason and ends up getting the 403: Error:

403 Client Error: Forbidden for url: https://download.pytorch.org/whl/cpu/mypy

As mentioned in this ticket, https://github.com/python-poetry/poetry/issues/4704 this is a known issue. However, amongst all possible ways to address this issue, this solution of using secondary sources seems to be the ideal fix for the issue in question.

As a short-term interim, I have also tried platform and version-specific settings. This would work fine if PyTorch was my leaf dependency. Because my setup involves using PyTorch, Torchvision, and Pytorch lightening. Because more dependencies rely on PyTorch, just specifying torch wheels in toml fails to solve the dependency: toml setting

torch = [
    { url="https://download.pytorch.org/whl/cu113/torch-1.11.0%2Bcu113-cp37-cp37m-linux_x86_64.whl", python=">=3.7,<3.8", markers="sys_platform == 'linux'"},
    { url="https://download.pytorch.org/whl/cu113/torch-1.11.0%2Bcu113-cp38-cp38-linux_x86_64.whl", python=">=3.8,<3.9", markers="sys_platform == 'linux'"},
    { url="https://download.pytorch.org/whl/cu113/torch-1.11.0%2Bcu113-cp39-cp39-linux_x86_64.whl", python=">=3.9,<3.10", markers="sys_platform == 'linux'"},
    { version = "=1.11.0", markers = "sys_platform == 'darwin' or sys_platform == 'win32'" },
]
torchvision = [
    { url="https://download.pytorch.org/whl/cu113/torchvision-0.12.0%2Bcu113-cp37-cp37m-linux_x86_64.whl", python=">=3.7,<3.8", markers="sys_platform == 'linux'"},
    { url="https://download.pytorch.org/whl/cu113/torchvision-0.12.0%2Bcu113-cp38-cp38-linux_x86_64.whl", python=">=3.8,<3.9", markers="sys_platform == 'linux'"},
    { url="https://download.pytorch.org/whl/cu113/torchvision-0.12.0%2Bcu113-cp39-cp39-linux_x86_64.whl", python=">=3.9,<3.10", markers="sys_platform == 'linux'"},
    { version = "=0.12.0", markers = "sys_platform == 'darwin' or sys_platform == 'win32'" },
]

Error:

  SolverProblemError

  Because torchvision (0.12.0+cu113) depends on torch (1.11.0)
   and XXXX-app depends on torch (1.11.0+cu113), torchvision is forbidden.
  So, because XXXX-app depends on torchvision (0.12.0+cu113), version solving failed.

I have been in knots with this one, particularly because there are so many issues open around this issue: https://github.com/python-poetry/poetry/issues/2543 https://github.com/python-poetry/poetry/issues/4231 https://github.com/python-poetry/poetry/issues/3855 https://github.com/python-poetry/poetry/issues/2613 https://github.com/python-poetry/poetry/issues/4704 https://github.com/python-poetry/poetry/issues/2339

The only solution that works cross-platform is https://github.com/nat-n/poethepoet but that is not a great solution either (not lining up with lock file, not using same cache etc, the need for additional pip run!). It would be great if we can fix this issue.

(this is a duplicate comment from https://github.com/python-poetry/poetry/issues/4704#issuecomment-1109465915, posting again as this is very relevant ticket again)

[abn]

Thanks for the detailed post @suneeta-mall. I want to try and break out targetted improvements to help with the pytorch use case being better supported. Let me try and addres a few things in your post with that intent.

but the default false is not recognized for some reason and ends up getting the 403

This is expected in that default = false is the default for legacy sources. The way the default setting works is more like "disable PyPI", ie. if set to true the project disables PyPI and makes it the "fallback" among all leacy sources (govered by secondary = true in your case). Personally, I think this need to reworked to make it clearer in Poetry.

Also see https://github.com/python-poetry/poetry/issues/4704#issuecomment-1111380597.

Regarding the local build tag resolving issue, the I suspect this is the same as being taked about in https://github.com/python-poetry/poetry/issues/4729#issuecomment-1110930059. That, in theory atleast, should fix this issue.

From the Poetry side once the local tag solving is fixed, I suspect things will improve. If that is not the case please do let me know.

For those working with the PyTorch community, if you can work with the PyTorch team to get the following added/fixed, your experience when using PyTorch with Poetry might be improved.

1. The lack of file hashes being exposed in the repository means that Poetry has to download each artifact unnecessarily. As per PEP 503. this can be appended to the file URL as #<hashname>=<hashvalue>. This can be done easily by the index admins I beleive.
2. While fixing (1) will reduce the number of required downloads, files still will be downloaded again when there is a need to inspect metadata (ie. Requires-Dist etc. from the wheel). This can be avoided too if PEP 658 gets implemented. I suspect this will need to also be supported by Poetry as well.
3. Would be great if requesting a non-existant resource returns a 404 instead of 403. If I understand correct, this is because they have not set s3:ListBucket permission for public users (assuming they use S3 for this).

PS: One could in theory generate and host files required for 1 and 2 using a CI/CD job + vercel/fastly etc with some packaging code if so motivated. The links can still retain the upstream file link, but with sha256 appended and a new .metadata file generated and served.

[suneeta-mall]

Thanks, @abn for the detailed info.

I have tried with poetry version 1.2.0b1 and wheels URL, and that was no joy as well. I did not get the solver error as mentioned in my earlier comment but the solving was indefinite 33947.4s and counting.

I have also raised the issue with Pytorch https://github.com/pytorch/pytorch/issues/76557 in line with your recommendation.

[tgolsson]

@suneeta-mall Was that infinite resolve with a clear cache? A colleague had issues with resolve never finishing but clearing the cache improved the situation. #5442 and #5451 are targeting specific other issues we've seen with using torch from Poetry which lead to very long resolve times.

[suneeta-mall]

@tgolsson yeah infinite resolve with clear cache on fresh docker build. Thanks for the PRs, the numbers look promising. In this case, the I left the resolve to go through overnight just for the fun of it and it continued.

Following has been my core of my change in toml b/w working and non-working [infinite resolve] copy, on version 1.2.0b1. Earlier version Not working snippet would give me resolve error as mentioned above:

Not working snippet

torch = [
    { url="https://download.pytorch.org/whl/cu113/torch-1.11.0%2Bcu113-cp37-cp37m-linux_x86_64.whl", python=">=3.7,<3.8", markers="sys_platform == 'linux'"},
    { url="https://download.pytorch.org/whl/cu113/torch-1.11.0%2Bcu113-cp38-cp38-linux_x86_64.whl", python=">=3.8,<3.9", markers="sys_platform == 'linux'"},
    { url="https://download.pytorch.org/whl/cu113/torch-1.11.0%2Bcu113-cp39-cp39-linux_x86_64.whl", python=">=3.9,<3.10", markers="sys_platform == 'linux'"},
    { version = "=1.11.0", markers = "sys_platform == 'darwin' or sys_platform == 'win32'" },
]
torchvision = [
    { url="https://download.pytorch.org/whl/cu113/torchvision-0.12.0%2Bcu113-cp37-cp37m-linux_x86_64.whl", python=">=3.7,<3.8", markers="sys_platform == 'linux'"},
    { url="https://download.pytorch.org/whl/cu113/torchvision-0.12.0%2Bcu113-cp38-cp38-linux_x86_64.whl", python=">=3.8,<3.9", markers="sys_platform == 'linux'"},
    { url="https://download.pytorch.org/whl/cu113/torchvision-0.12.0%2Bcu113-cp39-cp39-linux_x86_64.whl", python=">=3.9,<3.10", markers="sys_platform == 'linux'"},
    { version = "=0.12.0", markers = "sys_platform == 'darwin' or sys_platform == 'win32'" },
]

Working snippet

torch = "^1.11.0"
torchvision = "^0.12.0"

[PieterBlomme]

Note that this is largely because most torchvision wheels, including the oens on download.pytorch.org have a dependency on "torch", while they should have a dependency on a pinned version.
eg. https://download.pytorch.org/whl/cu113/torchvision-0.12.0%2Bcu113-cp37-cp37m-linux_x86_64.whl depends on torch==1.11.0 but should depend on torch==1.11.0+cu113

If you build torchvision from source you can set the proper torch dependency as follows:

• First install torch==1.11.0+cu113
• export PYTORCH_VERSION=1.11.0+cu113
• export BUILD_VERSION==0.12.0+cu113 # optional, but this sets the proper tag on the torchvision build
• git clone -b release/$TORCHVISION_VERSION https://github.com/pytorch/vision /torchvision
• cd torchvision
• python setup.py bdist_wheel

If you create wheels with this procedure, you can just add a dependency to torchvision in your pyproject.toml and no special tricks are needed anymore to make torch work well with Poetry. (Ofcourse you need to make sure Poetry has access to the wheels, either on a Devpi server or in a local folder)

[cateseale]

I was hopeful to try @PieterBlomme 's suggestion building torchvision from source as above, but no joy still...

Updating dependencies
Resolving dependencies... (623.5s)

Because torchvision (0.12.0+cu113) depends on torch (1.11.0+cu113)
 and my_package depends on torch (1.11.0+cu113), torchvision is forbidden.
So, because my_package depends on torchvision (0.12.0+cu113), version solving failed.

[cateseale]

Finally found a setup that at least is able to resolve. Using a torchvision wheel built from source as @PieterBlomme suggests, adding the secondary source for torch, and switching to poetry 0.12.

python = "^3.9"
torch = { version = "1.11.0+cu113", source = "pytorch" }
torchvision = {file = "/home/cate/git/torchvision/dist/torchvision-0.12.0+cu113-cp39-cp39-linux_x86_64.whl"}

[[tool.poetry.source]]
    name = "pytorch"
    url = "https://download.pytorch.org/whl/cu113"
    secondary = true

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[IWillPull]

I've also installed 1.2.0b1, but the solution proposed by @catesale is just too slow. Poetry cache grows to over 15GB of just torch cache...

I've added explicit sources for each package as pypi, but that does not work as it still tries to download everything from Pytorch repository.

Here's my config:

[tool.poetry.dependencies]
    python = "^3.10"
    fastapi = {version= '^0.75.1', source = "pypi"}
    uvicorn = {version= '^0.17.6', source = "pypi"}
    gunicorn = {version= '^20.1.0', source = "pypi"}
    requests = {version= "^2.27.1", source = "pypi"}
    torch = { version = '1.11.0+cu113', source = "pytorch" }

[tool.poetry.dev-dependencies]
    pre-commit = {version= '2.18.1', source = "pypi"}
    pytest = {version= '', source = "pypi"}

[[tool.poetry.source]]
    name = "pytorch"
    url = "https://download.pytorch.org/whl/cu113"
    secondary = true
    default = false


[build-system]
    requires = ["poetry-core>=1.0.0"]
    build-backend = 'poetry.masonry.api'

Output:

Creating virtualenv gpu-test-2-9TtSrW0h-py3.10 in /root/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies...
<debug>pytorch:</debug> Authorization error accessing https://download.pytorch.org/whl/cu113/virtualenv/
<debug>pytorch:</debug> Authorization error accessing https://download.pytorch.org/whl/cu113/toml/
<debug>pytorch:</debug> Authorization error accessing https://download.pytorch.org/whl/cu113/pyyaml/
<debug>pytorch:</debug> Authorization error accessing https://download.pytorch.org/whl/cu113/nodeenv/
<debug>pytorch:</debug> Authorization error accessing https://download.pytorch.org/whl/cu113/identify/
<debug>pytorch:</debug> Authorization error accessing https://download.pytorch.org/whl/cu113/cfgv/
<debug>pytorch:</debug> Authorization error accessing https://download.pytorch.org/whl/cu113/setuptools/
<debug>pytorch:</debug> Authorization error accessing https://download.pytorch.org/whl/cu113/h11/
<debug>pytorch:</debug> Authorization error accessing https://download.pytorch.org/whl/cu113/click/
<debug>pytorch:</debug> Authorization error accessing https://download.pytorch.org/whl/cu113/asgiref/
...

[TCherici]

@cateseale

switching to poetry 0.12

Don't you mean poetry 1.2.0b1?

[fedebotu]

Ehi there, just wanted to share my 50 cents since I may have found a working solution that automatically installs the CUDA version of PyTorch based on your configuration :) Note that this does not work by just running poetry install but by using an additional task with poe-the-poet and light-the-torch. Here is a sample configuration:

[tool.poetry.dependencies]
torch = "*"
poethepoet = "*"

[tool.poe.tasks]
install-ltt = "python3 -m pip install light-the-torch"
run-ltt = "python3 -m light_the_torch install --upgrade torch torchaudio torchvision"
autoinstall-torch-cuda = ["install-ltt", "run-ltt"]

Instructions:

1. Run the installation via poetry install
2. Then run the task with poetry run poe autoinstall-torch-cuda

[vikigenius]

I think the authorization issues should probably be fixed once https://github.com/python-poetry/poetry/pull/5442 is merged

[dimbleby]

The problem originally reported here is fixed as at https://github.com/python-poetry/poetry-core/pull/433/ ie this resolves just fine:

torch = {url = "https://download.pytorch.org/whl/cu111/torch-1.9.0%2Bcu111-cp38-cp38-linux_x86_64.whl"}
torchvision = {url = "https://download.pytorch.org/whl/cu111/torchvision-0.10.0%2Bcu111-cp38-cp38-linux_x86_64.whl"}

this is a long and confusing thread and I am unsure what other things might have become muddled into it: suggest close this (on the grounds that the actually reported problem is solved) and if other tickets are needed then raise them.

[sharvil]

In case it's helpful, here are some experiments I ran on two of my machines (Ubuntu 20.04.4). I've found that Config 1 works with 1.2.0b2 but not 1.2.0b3 (regression?). Config 2 is identical to Config 1 except it replaces the exact version requirement with a caret requirement.

Config 1

Config 1

[tool.poetry]
name = "test-pep-404"
version = "0.1.0"
description = ""
authors = [""]

[tool.poetry.dependencies]
python = "~3.9"
torch = { version = "=1.12.1+cu113", source = "torch" }
torchaudio = { version = "=0.12.1+cu113", source = "torch" }

[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/cu113"
secondary = true
default = false

[build-system]
requires = ["poetry_core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Poetry 1.2.0b2 - success

$ poetry install
Updating dependencies
Resolving dependencies... (1.3s)Source (torch): Authorization error accessing https://download.pytorch.org/whl/cu113/typing-extensions/
Resolving dependencies... (1.4s)

Writing lock file

Package operations: 2 installs, 0 updates, 0 removals

  • Installing torch (1.12.1+cu113)
  • Installing torchaudio (0.12.1+cu113)

Poetry 1.2.0b3 - fail

$ poetry install
Updating dependencies
Resolving dependencies... (1.0s)

[BUG] not torch (==1.12.1+cu113) is not satisfied.

Config 2

Config 2

name = "test-pep-404"
version = "0.1.0"
description = ""
authors = [""]

[tool.poetry.dependencies]
python = "~3.9"
torch = { version = "^1.12.1+cu113", source = "torch" }
torchaudio = { version = "^0.12.1+cu113", source = "torch" }

[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/cu113"
secondary = true
default = false

[build-system]
requires = ["poetry_core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Poetry 1.2.0b2 - fail

$ poetry install
Updating dependencies
Resolving dependencies... (1.0s)

Because no versions of torchaudio match >0.12.1+cu113,<0.13.0
 and torchaudio (0.12.1+cu113) depends on torch (1.12.1), torchaudio (>=0.12.1+cu113,<0.13.0) requires torch (1.12.1).
So, because test-pep-404 depends on both torch (^1.12.1+cu113) and torchaudio (^0.12.1+cu113), version solving failed.

Poetry 1.2.0b3 - fail

$ poetry install
Updating dependencies
Resolving dependencies... (1.1s)

Because no versions of torchaudio match >0.12.1+cu113,<0.13.0
 and torchaudio (0.12.1+cu113) depends on torch (1.12.1), torchaudio (>=0.12.1+cu113,<0.13.0) requires torch (1.12.1).
So, because test-pep-404 depends on both torch (^1.12.1+cu113) and torchaudio (^0.12.1+cu113), version solving failed.

[dimbleby]

In case it's helpful ...

It isn't! One of the following is true

• this is the same as the original, in which case it's already fixed in the most recent poetry / poetry-core code
• this is different, in which case it doesn't belong here: you should check whether it's already fixed on more recent code and raise a new issue if needed

either way, this issue should be closed

[sharvil]

You're welcome to open a new issue if you'd like and close this one if you can. I shared my experience for those who, like me, have been struggling to get PyTorch and related libraries working with released versions of Poetry and need a working solution today.