Poetry / PEP 517 build support

[hongbo-miao]

trafficstars

I am trying to install by Poetry. However, it seems this library failed to be installed because of PEP 517 build support.

pyproject.toml:

[tool.poetry]
name = "my-app"
version = "1.0.0"
description = ""
authors = ["Hongbo Miao"]

[tool.poetry.dependencies]
python = "3.8.x"
torch = "1.13.1"
torch-cluster = "1.6.1"
torch-geometric = "2.3.1"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

➜ poetry env use 3.8
➜ poetry lock --no-update
➜ poetry install --no-root
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing torch-cluster (1.6.1): Pending...
  • Installing torch-cluster (1.6.1): Preparing...
  • Installing torch-cluster (1.6.1): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  Traceback (most recent call last):
    File "/opt/homebrew/Cellar/poetry/1.5.1_1/libexec/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/opt/homebrew/Cellar/poetry/1.5.1_1/libexec/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/opt/homebrew/Cellar/poetry/1.5.1_1/libexec/lib/python3.11/site-packages/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/var/folders/22/ntjwd5dx691gvkktkspl0f_00000gq/T/tmp3ud_ljj_/.venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/var/folders/22/ntjwd5dx691gvkktkspl0f_00000gq/T/tmp3ud_ljj_/.venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
      self.run_setup()
    File "/var/folders/22/ntjwd5dx691gvkktkspl0f_00000gq/T/tmp3ud_ljj_/.venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 487, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/var/folders/22/ntjwd5dx691gvkktkspl0f_00000gq/T/tmp3ud_ljj_/.venv/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 8, in <module>
  ModuleNotFoundError: No module named 'torch'
  

  at /opt/homebrew/Cellar/poetry/1.5.1_1/libexec/lib/python3.11/site-packages/poetry/installation/chef.py:147 in _prepare
      143│ 
      144│                 error = ChefBuildError("\n\n".join(message_parts))
      145│ 
      146│             if error is not None:
    → 147│                 raise error from None
      148│ 
      149│             return path
      150│ 
      151│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with torch-cluster (1.6.1) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "torch-cluster (==1.6.1)"'.

make: *** [poetry-install] Error 1

I further verified based on the method in the log by

➜ pip wheel --use-pep517 "torch-cluster (==1.6.1)"
Collecting torch-cluster==1.6.1
  Downloading torch_cluster-1.6.1.tar.gz (53 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.8/53.8 kB 988.6 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      Traceback (most recent call last):
        File "/Users/hongbo-miao/.pyenv/versions/3.11.1/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/Users/hongbo-miao/.pyenv/versions/3.11.1/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/hongbo-miao/.pyenv/versions/3.11.1/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/22/ntjwd5dx691gvkktkspl0f_00000gq/T/pip-build-env-tlczvsao/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/22/ntjwd5dx691gvkktkspl0f_00000gq/T/pip-build-env-tlczvsao/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/22/ntjwd5dx691gvkktkspl0f_00000gq/T/pip-build-env-tlczvsao/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 488, in run_setup
          self).run_setup(setup_script=setup_script)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/22/ntjwd5dx691gvkktkspl0f_00000gq/T/pip-build-env-tlczvsao/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 8, in <module>
      ModuleNotFoundError: No module named 'torch'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Any guide would be appreciate, thanks! 😃

[rusty1s]

There is no clear workaround for this. You will need PyTorch installed to actually build torch-cluster. The only way to resolve this is by installing from the wheels in https://data.pyg.org/whl, which do not have this restriction as they are pre-built.

[QuentinSoubeyranAqemia]

I'm running into this issue as well.

Is there anything special in torch-cluster preventing the use of the legacy setup_requires argument of setuptools.setup(), or the modern build-system.requires entry in pyproject.toml, to specify build-time dependencies? Build-time dependencies have been part of setuptools since before PEP 517, and that PEP further standardized those issues in the pyproject.toml file.

Setuptools docs for build-time dependencies

[QuentinSoubeyranAqemia]

I tried installing wheels directly from https://data.pyg.org/whl, to no avail: poetry doesn't find anything in the top-level URL nor the torch-specific URL, even though poetry tries to support single-page repository. I'm not sure the pages follow PEP 503 which seems to be the relevant specification.

I guess I could list each wheel manually, for each python version and platform using environment constraints, but that sorts of defeats the purpose of dependency managers.

[rusty1s]

There is no particular reason we haven't made the change to pyproject.toml yet - mostly because this repository is no longer in active development since we moved the core parts of it to pyg-lib.

Regarding [build-system] (and please correct me if I am wrong): We cannot easily specify torch here as this would likely install a wrong CUDA version. That's why we generally assume that torch is already installed on your system (expect for when installing from wheels).

Regarding poetry: I am not a poetry user myself, so this has been not a top-priority for me. pip and conda should run fine though. I am happy to include poetry support here, but would probably need some help and guidance on this one.

[QuentinSoubeyranAqemia]

Regarding PEP 517 support

I believe this applies to all packages of the torch-geometric family, which IIRC all needs to be compiled against an exact version of pytorch. So this is possibly more wide-ranging topic than just this repo.

---

I understand the switch to pyproject.toml from setuptools is a large one, especially if this repo is no longer in active development. I think it is possible to support PEP 517 at low-ish cost by:

• using the setup_requires argument of setuptools.setup(), to avoid switching to pyproject.toml in a not-in-active-development project
• Using the arbitrary equality operator === in setup_requires version constraints to force an exact pytorch version, including the CUDA specifier +cu112 at the end

Though I have not tested it yet. I might manage to find some time to try and test that.

[QuentinSoubeyranAqemia]

Poetry installation

I did manage to install torch-geometric, torch-cluster and torch-scatter from the pre-built wheels. My mistake yesterday was to try to install torch-geometric from https://data.pyg.org/whl while it is not there.

For reference, I used the following commands:

poetry add torch==2.1.1
poetry source add -p explicit pytorch-geometric https://data.pyg.org/whl/torch-2.1.0+cu121.html
poetry add --source pytorch-geometric torch-scatter torch-cluster
poetry add torch-geometric

Which generated the following pyproject.toml:

[[tool.poetry.source]]
name = "pytorch-geometric"
url = "https://data.pyg.org/whl/torch-2.1.0+cu121.html"
priority = "explicit"

[tool.poetry.dependencies]
python = ">=3.8, <3.13"
torch = {version = "2.1.1", source = "pypi"}
torchvision = {version = "^0.16.1", source = "pypi"}
torchaudio = {version = "^2.1.1", source = "pypi"}
torch-geometric = {version = "2.4.0", source="pypi"}
torch-cluster = {version = "^1.6.3+pt21cu121", source = "pytorch-geometric"}
torch-scatter = {version = "^2.1.2+pt21cu121", source = "pytorch-geometric"}

[github-actions[bot]]

This issue had no activity for 6 months. It will be closed in 2 weeks unless there is some new activity. Is this issue already resolved?

[QuentinSoubeyranAqemia]

This issue is not solve AFAIK and still makes it difficult to install pytorch_cluster

[stefanplaza]

Quentin, when I use your method, I have no problem installing torch_geometric and torch_geometric_temporal, however when I try to import them, my kernel crashes

[stefanplaza]

the problem comes from torch_sparse I think