How do I lock `torch-scatter`?

[NellyWhads]

Here's a relevant example project toml in which I attempt to lock torch-scatter as a dependency which only installs on a specific platform:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "test_ml"
version = "0.0.0"
requires-python = ">=3.8,<3.11"

dependencies = [
  "torch==2.1.0",
  "torch==2.1.0+cu118 ; sys_platform == 'linux'",
  "torchvision==0.16.0",
  "torchvision==0.16.0+cu118 ; sys_platform == 'linux'",
  #   "torch-scatter==2.1.2 ; sys_platform == 'linux'",
  #   "torch-scatter==2.1.2+pt21 ; sys_platform == 'linux'",
  #   "torch-scatter==2.1.2+cu118 ; sys_platform == 'linux'",
  "torch-scatter==2.1.2+pt21cu118 ; sys_platform == 'linux'",
]

[tool.uv.sources]
torch = [{ index = "torch-cu118", marker = "sys_platform != 'darwin'" }]
torchvision = [{ index = "torch-cu118", marker = "sys_platform != 'darwin'" }]
torch-scatter = [
  # { index = "torch-scatter-torch210", marker = "sys_platform == 'darwin'" },
  { index = "torch-scatter-torch210-cu118", marker = "sys_platform == 'linux'" },
]

[[tool.uv.index]]
name = "torch-scatter-torch210"
url = "https://data.pyg.org/whl/torch-2.1.0+cpu.html"
explicit = true

[[tool.uv.index]]
name = "torch-scatter-torch210-cu118"
url = "https://data.pyg.org/whl/torch-2.1.0+cu118.html"
explicit = true

[[tool.uv.index]]
name = "torch-cu118"
url = "https://download.pytorch.org/whl/cu118"
explicit = true

[tool.uv]
environments = ["sys_platform == 'darwin'", "sys_platform == 'linux'"]

I can't seem to figure out how to lock torch-scatter, let alone build/install it. Any assistance would be apprecaited.

I looked through the previous github issues and don't see anything related to the lock in particular. I'm hoping to set up this config to support both darwin and linux, however, the dependency can be ignored on darwin hosts.

[FishAlchemist]

Based on my knowledge, UV does not yet provide support for utilizing URLs from find-links as a source within the [[tool.uv.index]] configuration. Therefore, your only option right now is to configure find-links to meet your requirements. https://docs.astral.sh/uv/reference/settings/#find-links Comments mentioning unsupported this features: https://github.com/astral-sh/uv/issues/8717#issuecomment-2451876796

As for whether [[tool.uv.index]] will support find-links in the future, we'll have to wait and see what the UV team decides.

[NellyWhads]

Hmm, could you provide a quick example of how to do that in this situation? Is find-links url-specific? Would it significantly simplify the versions since I could use plain torch and torchvision names?

[FishAlchemist]

@NellyWhads I think this can be used as a reference. I'm not sure if there's a better way to do it, since I don't usually use find-links.

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "test_ml"
version = "0.0.0"
requires-python = ">=3.8,<3.11"
dependencies = [
  "torch==2.1.0",
  "torch==2.1.0+cu118 ; sys_platform == 'linux'",
  "torchvision==0.16.0",
  "torchvision==0.16.0+cu118 ; sys_platform == 'linux'",
  #   "torch-scatter==2.1.2 ; sys_platform == 'linux'",
  #   "torch-scatter==2.1.2+pt21 ; sys_platform == 'linux'",
  #   "torch-scatter==2.1.2+cu118 ; sys_platform == 'linux'",
  "torch-scatter==2.1.2+pt21cu118 ; sys_platform == 'linux'",
]

[tool.uv.sources]
torch = [{ index = "torch-cu118", marker = "sys_platform != 'darwin'" }]
torchvision = [{ index = "torch-cu118", marker = "sys_platform != 'darwin'" }]
#torch-scatter = [
#  # { index = "torch-scatter-torch210", marker = "sys_platform == 'darwin'" },
#  { index = "torch-scatter-torch210-cu118", marker = "sys_platform == 'linux'" },
#]

#[[tool.uv.index]]
#name = "torch-scatter-torch210"
#url = "https://data.pyg.org/whl/torch-2.1.0+cpu.html"
#explicit = true

#[[tool.uv.index]]
#name = "torch-scatter-torch210-cu118"
#url = "https://data.pyg.org/whl/torch-2.1.0+cu118.html"
#explicit = true

[[tool.uv.index]]
name = "torch-cu118"
url = "https://download.pytorch.org/whl/cu118"
explicit = true

[tool.uv]
environments = ["sys_platform == 'darwin'", "sys_platform == 'linux'"]
find-links = [
  "https://data.pyg.org/whl/torch-2.1.0+cpu.html",
  "https://data.pyg.org/whl/torch-2.1.0+cu118.html"
]

[NellyWhads]

Thanks @FishAlchemist I actually simplified it further to this:

pyproject.toml

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "test_ml"
version = "0.0.0"
requires-python = ">=3.8,<3.11"

dependencies = [
  "torch==2.1.0",
  "torchvision==0.16.0",
  # "torch-scatter==2.1.2 ; sys_platform == 'darwin'",
  "torch-scatter==2.1.2+pt21cu118 ; sys_platform == 'linux'",
]

[tool.uv.sources]
torch = [{ index = "torch-cu118", marker = "sys_platform == 'linux'" }]
torchvision = [{ index = "torch-cu118", marker = "sys_platform == 'linux'" }]

[[tool.uv.index]]
name = "torch-cu118"
url = "https://download.pytorch.org/whl/cu118"
explicit = true

[tool.uv]
environments = ["sys_platform == 'darwin'", "sys_platform == 'linux'"]
find-links = [
  # "https://data.pyg.org/whl/torch-2.1.0+cpu.html",
  "https://data.pyg.org/whl/torch-2.1.0+cu118.html",
]

My last question; how I can lock the darwin version as well? The CPU index does have wheels with no extra version specifiers which allow for installation of the package on macosx (ie. 2.1.2 instead of 2.1.2+pt21cpu). Pip install of course works, but I can't seem to tell uv to look explicitly for the version without the extra tags when on the darwin platform.

[NellyWhads]

Is there some sort of "exact match" specifier I'm missing?

[FishAlchemist]

@charliermarsh Since 0.5.0, it seems quite difficult to lock dependencies to a specific version without local version tags on macOS, based on dependency requirements. Is this the expected behavior starting from 0.5.0?

requires-python = ">=3.8,<3.11"
dependencies = [
  "torch==2.1.0 ; sys_platform == 'darwin'",
  "torchvision==0.16.0 ; sys_platform == 'darwin'",
  "torch==2.1.0+cu118 ; sys_platform == 'linux'",
  "torchvision==0.16.0+cu118 ; sys_platform == 'linux'",
  "torch-scatter==2.1.2; sys_platform == 'darwin'",
  "torch-scatter==2.1.2+pt21cu118 ; sys_platform == 'linux'",
]

[tool.uv.sources]
torch = [{ index = "torch-cu118", marker = "sys_platform == 'linux'" }]
torchvision = [{ index = "torch-cu118", marker = "sys_platform == 'linux'" }]

[[tool.uv.index]]
name = "torch-cu118"
url = "https://download.pytorch.org/whl/cu118"
explicit = true

[tool.uv]
environments = ["sys_platform == 'darwin'", "sys_platform == 'linux'"]
find-links = [
  "https://data.pyg.org/whl/torch-2.1.0+cpu.html",
  "https://data.pyg.org/whl/torch-2.1.0+cu118.html",
]

uv.lock for uv 0.4.29 (85f9a0d0e 2024-10-30)

[[package]]
name = "torch-scatter"
version = "2.1.2"
source = { registry = "https://data.pyg.org/whl/torch-2.1.0+cpu.html" }
resolution-markers = [
    "sys_platform == 'darwin'",
]
wheels = [
    { url = "https://data.pyg.org/whl/torch-2.1.0%2Bcpu/torch_scatter-2.1.2-cp310-cp310-macosx_11_0_x86_64.whl" },
    { url = "https://data.pyg.org/whl/torch-2.1.0%2Bcpu/torch_scatter-2.1.2-cp38-cp38-macosx_11_0_x86_64.whl" },
    { url = "https://data.pyg.org/whl/torch-2.1.0%2Bcpu/torch_scatter-2.1.2-cp39-cp39-macosx_11_0_x86_64.whl" },
]

[[package]]
name = "torch-scatter"
version = "2.1.2+pt21cu118"
source = { registry = "https://data.pyg.org/whl/torch-2.1.0+cu118.html" }
resolution-markers = [
    "sys_platform == 'linux'",
]
wheels = [
    { url = "https://data.pyg.org/whl/torch-2.1.0%2Bcu118/torch_scatter-2.1.2%2Bpt21cu118-cp310-cp310-linux_x86_64.whl" },
    { url = "https://data.pyg.org/whl/torch-2.1.0%2Bcu118/torch_scatter-2.1.2%2Bpt21cu118-cp38-cp38-linux_x86_64.whl" },
    { url = "https://data.pyg.org/whl/torch-2.1.0%2Bcu118/torch_scatter-2.1.2%2Bpt21cu118-cp39-cp39-linux_x86_64.whl" },
]

uv.lock for uv 0.5.0 (8d665267c 2024-11-07)

[[package]]
name = "torch-scatter"
version = "2.1.2+pt21cu118"
source = { registry = "https://data.pyg.org/whl/torch-2.1.0+cu118.html" }
wheels = [
    { url = "https://data.pyg.org/whl/torch-2.1.0%2Bcu118/torch_scatter-2.1.2%2Bpt21cu118-cp310-cp310-linux_x86_64.whl" },
    { url = "https://data.pyg.org/whl/torch-2.1.0%2Bcu118/torch_scatter-2.1.2%2Bpt21cu118-cp38-cp38-linux_x86_64.whl" },
    { url = "https://data.pyg.org/whl/torch-2.1.0%2Bcu118/torch_scatter-2.1.2%2Bpt21cu118-cp39-cp39-linux_x86_64.whl" },
]

[NellyWhads]

Hey folks. any chance we can re-visit this? Coming up against it as a blocker in a machine learning mono-repo

[NellyWhads]

Another option to resolve cases like this would be to allow uv indices to use the find-links functionality directly. This way, for the CPU-only groups, indices could be used without the find-links feature, whereas for GPU groups/indices, it can be toggled on.

The central find-links setting does not allow for resolution of package with and without tags successfully as currently implemented.

[charliermarsh]

I think you want this:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "test_ml"
version = "0.0.0"
requires-python = ">=3.8,<3.11"

dependencies = [
  "torch==2.1.0",
  "torchvision==0.16.0",
  "torch-scatter==2.1.2 ; sys_platform != 'darwin'"
]

[tool.uv.sources]
torch = [
    { index = "torch-cu118", marker = "sys_platform != 'darwin'" },
    { index = "torch-cpu", marker = "sys_platform == 'darwin'" },
]
torchvision = [
    { index = "torch-cu118", marker = "sys_platform != 'darwin'" },
    { index = "torch-cpu", marker = "sys_platform == 'darwin'" },
]

[[tool.uv.index]]
name = "torch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true


[[tool.uv.index]]
name = "torch-cu118"
url = "https://download.pytorch.org/whl/cu118"
explicit = true

[tool.uv]
find-links = ["https://data.pyg.org/whl/torch-2.1.0+cu118.html"]

[charliermarsh]

On macOS:

test-ml v0.0.0
├── torch v2.1.0
│   ├── filelock v3.16.1
│   ├── fsspec v2024.12.0
│   ├── jinja2 v3.1.5
│   │   └── markupsafe v3.0.2
│   ├── networkx v3.4.2
│   ├── sympy v1.13.3
│   │   └── mpmath v1.3.0
│   └── typing-extensions v4.12.2
└── torchvision v0.16.0
    ├── numpy v2.2.1
    ├── pillow v11.0.0
    ├── requests v2.32.3
    │   ├── certifi v2024.12.14
    │   ├── charset-normalizer v3.4.1
    │   ├── idna v3.10
    │   └── urllib3 v2.3.0
    └── torch v2.1.0 (*)
(*) Package tree already displayed

On Linux:

test-ml v0.0.0
├── torch v2.1.0+cu118
│   ├── filelock v3.16.1
│   ├── fsspec v2024.12.0
│   ├── jinja2 v3.1.5
│   │   └── markupsafe v3.0.2
│   ├── networkx v3.4.2
│   ├── sympy v1.13.3
│   │   └── mpmath v1.3.0
│   ├── triton v2.1.0
│   │   └── filelock v3.16.1
│   └── typing-extensions v4.12.2
├── torch-scatter v2.1.2+pt21cu118
└── torchvision v0.16.0+cu118
    ├── numpy v2.2.1
    ├── pillow v11.0.0
    ├── requests v2.32.3
    │   ├── certifi v2024.12.14
    │   ├── charset-normalizer v3.4.1
    │   ├── idna v3.10
    │   └── urllib3 v2.3.0
    └── torch v2.1.0+cu118 (*)
(*) Package tree already displayed

[NellyWhads]

The solution you described is what I currently use, however, modern torch-scatter does support MacOS (example of the torch~=2.1.0 wheel source).

This issue is raised to find a way to lock working versions of torch-scatter for MacOS and Linux w/ Cuda 11.8 using pre-built wheels.

[charliermarsh]

I believe this does what you're describing:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "test_ml"
version = "0.0.0"
requires-python = ">=3.8,<3.11"

dependencies = [
  "torch==2.1.0",
  "torchvision==0.16.0",
  "torch-scatter==2.1.2"
]

[tool.uv.sources]
torch = [
    { index = "torch-cu118", marker = "sys_platform != 'darwin'" },
    { index = "torch-cpu", marker = "sys_platform == 'darwin'" },
]
torchvision = [
    { index = "torch-cu118", marker = "sys_platform != 'darwin'" },
    { index = "torch-cpu", marker = "sys_platform == 'darwin'" },
]

[[tool.uv.index]]
name = "torch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true


[[tool.uv.index]]
name = "torch-cu118"
url = "https://download.pytorch.org/whl/cu118"
explicit = true

[tool.uv]
find-links = ["https://data.pyg.org/whl/torch-2.1.0+cu118.html", "https://data.pyg.org/whl/torch-2.1.0+cpu.html"]

[charliermarsh]

It only includes the following wheels though since these are the only wheels published for macOS in your Python range:

wheels = [
    { url = "https://data.pyg.org/whl/torch-2.1.0%2Bcpu/torch_scatter-2.1.2-cp310-cp310-macosx_11_0_x86_64.whl" },
    { url = "https://data.pyg.org/whl/torch-2.1.0%2Bcpu/torch_scatter-2.1.2-cp38-cp38-macosx_11_0_x86_64.whl" },
    { url = "https://data.pyg.org/whl/torch-2.1.0%2Bcpu/torch_scatter-2.1.2-cp39-cp39-macosx_11_0_x86_64.whl" },
]

[NellyWhads]

I'll try that config out - I think it's working because of the order of the find-links urls, am I right? I was getting odd behavior when I tried similar configs above.

[charliermarsh]

Optimistically closing for now.

[NellyWhads]

Apologies - have not had the bandwidth to re-test this in my current repo. But running the isolated example seems to behave as described.