rapidsai
[FEA] Enable cuda-version dependency resolution for packages that depend on cucim or other rapids packages
Problem Statement
I want to create packages that depend on cuCIM or other RAPIDs packages. Through standard Python package dependencies, it would be very helpful to
- Have a correct compatible version pulled in automatically without hardcoding a specific CUDA version when it is not needed directly
- Conflicts, e.g. CUDA versions, are detected and an informative error provided
Example package installation with pip itkwasm-downsample-cucim.
Right now, all compatible CUDA dependencies and any other unrelated packages with CUDA dependencies need to be pre-installed explicitly by the user, who must know that they are installing compatible versions of everything (and their transitive dependencies are compatible). Or, a hard-coded cuda version is added as a dependency, which might conflict with other package dependencies.
A possible approach that might work (?) for pip, similar to the cuda-version used in conda, is to use the ability to have optional dependencies that are self-referential in the following way:
- cucim[cudefault] dependency depends on cucim[cu11], cucim[cu11] depends on cuda-version~=11, cupy[cu11]
- my-package-uses-cucim depends on cucim[cudefault]
- RAPIDs packages all periodically update [cudefault] dependencies on [cuXX] in harmony
cucim pyproject.toml:
[project]
name = "cucim"
[project.optional-dependencies]
cu11 = [
"cucim-cu11",
"cuda-version~=11",
]
cu12 = [
"cucim-cu12",
"cuda-version~=12",
]
cudefault = [
"cucim[cu11]",
]
Where cuda-version is a meta-package used to identify dependency conflicts. Ideally cuda-version is in both in conda and pip so both dependency resolvers pick up conflicts.
my-package-uses-cucim pyproject.toml:
[project]
name = "my-package-uses-cucim"
dependencies = [
"cucim[cudefault]"
]
Hey Matt, a cuda-version metapackage for pip is definitely something we'd like to implement, it just hasn't really risen that high on the priority list so far.
Your proposal with extras is interesting. There is some benefit to using extras to specify the version via a cucim metapackage as opposed to using separate packages the way we currently have (assuming the cuda-version metapackage were introduced), but not a big one The way I see it, there are three problems we want to solve.
Ensure consistent package installation
Consider the following two packages:
[project]
name = "cucim-cu11"
dependencies = [
"cuda-version~=11",
]
[project]
name = "cucim-cu12"
dependencies = [
"cuda-version~=12",
]
If we wanted to install cucim along with another RAPIDS package that was also CUDA-dependent and structured the same way, say cuml, the cuda-version metapackage would prevent us from doing something like pip install cucim-cu11 cuml-cu12 because those CUDA versions would be mismatched. That works just as well using two separate packages as via a package with extras.
Avoid dependent packages from needing to update the dependency name
If you have a package depending on cucim, then the cucim[cudefault] approach is a nice way to save you from needing to periodically bump your dependency list from cucim-cuX to cucim-cu{X+1}.
Allow version selection solely via cuda-version
With our conda packages, you can do something quite nice with cuda-version:
conda install cucim cuml cuda-version=12.0
Rather than specifying a version for every package, you can just specify the cuda-version package version and the conda solver will correctly figure out the versions of cuda-cudart and cuda-cccl to include. Ideally we'd like to replicate something like that for pip, but that will require a bit more thought. I also don't think that would benefit much from the extras approach.
Does this summary seem right to you? Basically the second item is the main one where extras give you a benefit AFAICT.
Related discussion in issue: https://github.com/rapidsai/build-planning/issues/28
Hi Vyas,
@vyasr Yes, you a right, the second point is the main benefit which could be solved with the current extras support in the ecosystem and the proposed extras usage:
Avoid dependent packages from needing to update the dependency name
but not a big one The way I see it,
It is a big one, and this importance comes from taking the perspective of someone who is developing a package that would like to depend on a RAPIDS package. Or, from someone who just wants to use that package and needing to know the details of the CUDA version or encountering conflicts is a major pain point.
https://peps.python.org/pep-0771/ would be the way that we probably want to support this eventually.