conda-forge
Local CUDA installation not being detected
Conda-forge documentation
- [x] I could not solve my problem using the conda-forge documentation.
Installed packages
conda install tensorflow
Environment info
active environment : base
active env location : /root/work/apps/miniforge3
shell level : 1
user config file : /root/.condarc
populated config files : /root/work/apps/miniforge3/.condarc
/root/.condarc
conda version : 24.11.3
conda-build version : not installed
python version : 3.12.8.final.0
solver : libmamba (default)
virtual packages : __archspec=1=zen4
__conda=24.11.3=0
__cuda=12.4=0
__glibc=2.35=0
__linux=6.8.0=0
__unix=0=0
base environment : /root/work/apps/miniforge3 (writable)
conda av data dir : /root/work/apps/miniforge3/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
package cache : /root/work/apps/miniforge3/pkgs
/root/.conda/pkgs
envs directories : /root/work/apps/miniforge3/envs
/root/.conda/envs
platform : linux-64
user-agent : conda/24.11.3 requests/2.32.3 CPython/3.12.8 Linux/6.8.0-51-generic ubuntu/22.04.4 glibc/2.35 solver/libmamba conda-libmamba-solver/24.9.0 libmambapy/1.5.11
UID:GID : 0:0
netrc file : None
offline mode : False
Issue
Installing tensorflow package results to conda pulling the same cuda version as that of the system install. It appears that conda is not detecting it and resulting to duplication.
note that paths have also been exported to no avail
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
export PATH=$CUDA_HOME/bin:$PATH
however normal pip install does not attempt to reinstall cuda toolkit and detects system cuda, so clearly this is a conda-forge issue not resolving library requirements.
This is expected behavior. Conda ships the entire CUDA Toolkit as Conda packages
This is expected behavior. Conda ships the entire CUDA Toolkit as Conda packages
pip does not have this issue as:
- pip does not force to install its own cuda toolkit
- the installed package detected and utilized the system installed cuda libraries like its
plug and play
The point to understand is that "detecting the system CUDA" is not design goal of how CUDA is distributed in conda-forge. It's not that conda can't see the system CUDA; it's not even looking. The only thing we take into account is the driver compatibility via the __cuda virtual package (which you can see in the conda info output).
There have been similar asks in the past (e.g. https://github.com/conda-forge/cudatoolkit-feedstock/issues/61) if you want to take a look.
Right
Since pip is a Python package manager, it can make the choice to not ship things that are not Python (like C/C++ libraries)
Conda is language agnostic. So users typically expect an install command to pull in everything they need (Python & C/C++)
So the general approach has been to broadly ship things as Conda packages. CUDA packages are merely a subset that are also handled this way. Though it is not unique to them