Consider switching to uv for dependency management (esp. PyTorch)
There are currently several development efforts ongoing that would need PyTorch as a dependency:
- #798
- #914
- #895
PyTorch is a bit special in that it provides different library variants for different kinds of hardware, especially GPUs. uv has special support to manage that, while Poetry doesn't.
Some discussion from https://github.com/NatLibFi/Annif/pull/914#issuecomment-3546400885
Here are some things that I think would be desirable:
- It should be possible to use Annif (with EBM) without installing GPU dependencies (even if it's slow). This would be especially desirable in the case of GitHub Actions CI because CI jobs run very often so they should be as lightweight as possible and complete as fast as possible.
- It should be possible to use different brands of GPUs, not just limited to NVIDIA/CUDA but also AMD/ROCm and possibly others (e.g. Vulkan) if PyTorch has support.
I realize that these may be difficult to achieve in our current way of managing dependencies. In my understanding, Poetry only has limited support for PyTorch variants making it difficult to implement flexible choices. I think
uvhas better support - see e.g. here. So we could consider switching touvif it helps in this area.
and https://github.com/NatLibFi/Annif/pull/914#issuecomment-3558849902 :
I must say I'm tempted by the special support for PyTorch that uv provides. It would e.g. allow specifying the PyTorch variant at install time, something like this:
uv pip install annif[ebm] --torch-backend=cu126
or
uv pip install annif[ebm] --torch-backend=cpu
If we want to make use of that for Annif itself, we would have to switch from Poetry to uv, which is probably not trivial but should be doable. We have already switched dependency management systems several times in Annif history: I think we started with pipenv, then switch to plain pip+venv, and more recently have been using Poetry.
Switching to uv would affect at least these parts of Annif:
- pyproject.toml: drop Poetry specific configuration, possible add some uv-specific config
- managing development installs (documented in README.md)
- GitHub Actions workflows
- wiki documentation (e.g. how to install optional dependencies)