cudf
cudf copied to clipboard
rapidsai
[QST] Why does cudf report a warning related to numba?
WARNING message:
(cuda) me@deskwin10:~/code$ python tmp.py 15:33:14 [39/1844]
/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/utils/_ptxcompiler.py:64: UserWarning: Error getting driver and runtime versions:
stdout:
stderr:
Traceback (most recent call last):
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 277, in ensure_initialized
self.cuInit(0)
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 327, in safe_cuda_api_call
self._check_ctypes_error(fname, retcode)
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 395, in _check_ctypes_error
raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [100] Call to cuInit results in CUDA_ERROR_NO_DEVICE
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 4, in <module> File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 292, in __getattr__
self.ensure_initialized() File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 281, in ensure_initialized
raise CudaSupportError(f"Error at driver init: {description}") numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: Call to cuInit results in CUDA_ERROR_NO_DEVICE (100)
Not patching Numba warnings.warn(msg, UserWarning)
The code looks like:
import numpy as np
import pandas as pd
import cudf.pandas
cudf.pandas.install()
def main():
df = pd.read_csv('my_file.csv.bz2', compression='bz2', index_col=0)
df = df[df.index < '2024-04-01'].copy()
return
if __name__ == '__main__':
main()
nvidia-smi has the following output:
(cuda) me@deskwin10:~/code$ nvidia-smi
Tue Jun 3 04:33:27 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.230.02 Driver Version: 560.94 CUDA Version: 12.6 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 2060 ... On | 00000000:01:00.0 Off | N/A |
| 0% 50C P8 5W / 175W | 10MiB / 8192MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
cudf version:
(cuda) me@deskwin10:~/code$ pip show cudf-cu12
Name: cudf-cu12
Version: 25.4.0
Summary: cuDF - GPU Dataframe
Home-page: https://github.com/rapidsai/cudf
Author: NVIDIA Corporation
Author-email:
License: Apache-2.0
Location: /usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages
Requires: cachetools, cuda-python, cupy-cuda12x, fsspec, libcudf-cu12, numba, numba-cuda, numpy, nvtx, packaging, pandas, pyarrow, pylibcudf-cu12, pynvjitlink-cu12, rich, rmm-cu12, typing_extensions
Required-by:
The platform is WSL Ubuntu 24.04 on WIN 10.
How can I fix it?
@blue-cat-whale Can you tell us a bit more on how are you creating your environment?
@blue-cat-whale Can you tell us a bit more on how are you creating your environment?
cudf is installed via pip install --extra-index-url=https://pypi.nvidia.com "cudf-cu12==25.4.*" and the numba version is 0.61.2, numba-cuda==0.4.0, pandas=2.2.3, pip=25.1.1.
I use virtualenvwrapper for virtual env management and the virtual env is created (and activated) via mkvirtualenv cuda; workon cuda.
`
I'll transfer this into a cudf issue. I think there are some similar issues reported in the past, I will look for those and link them here.
Let's try the debugging steps listed here: https://github.com/rapidsai/cudf/issues/17590#issuecomment-2578591770
Can you share the output of this command?
import numba.cuda
print(numba.cuda.cudadrv.driver.locate_driver_and_loader())
print(numba.cuda.cudadrv.driver.driver.cuInit(0))
One other note, which is probably unrelated to the root cause here: you have to call cudf.pandas.install() before importing pandas. Your code should be modified like this:
# Set up cudf.pandas first
import cudf.pandas
cudf.pandas.install()
# Then import pandas
import numpy as np
import pandas as pd
A few other notes that may be helpful in debugging:
Can you share more about how you installed the CUDA Toolkit? On GeForce cards, forward compatibility is not supported. That means that your driver must be at least as new as the CUDA Toolkit version you use. If you're using CUDA Toolkit 12.9 (the latest), you also need to update your driver to the latest. I see you have driver 560.94 and the latest version for Windows with RTX 2060 is 576.52.
On WSL, the driver you get should be from the host Windows system -- you do not need a driver in the Linux OS. If you have a driver installed in your WSL instance, uninstall it first!
On WSL, you will need the nvidia-container-toolkit to be installed if you run in a Docker container. You can follow the instructions here: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
The short list of commands for that is:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
# Don't forget to set up Docker!
sudo nvidia-ctk runtime configure --runtime=docker
Let's try the debugging steps listed here: #17590 (comment)
Can you share the output of this command?
import numba.cuda
print(numba.cuda.cudadrv.driver.locate_driver_and_loader())
print(numba.cuda.cudadrv.driver.driver.cuInit(0))
One other note, which is probably unrelated to the root cause here: you have to call
cudf.pandas.install()before importingpandas. Your code should be modified like this:Set up cudf.pandas first
import cudf.pandas cudf.pandas.install()
Then import pandas
import numpy as np import pandas as pd
- The output of
numba.cuda.
>>> import numba.cuda
>>> print(numba.cuda.cudadrv.driver.locate_driver_and_loader())
(<class 'ctypes.CDLL'>, ['libcuda.so', 'libcuda.so.1', '/usr/lib/libcuda.so', '/usr/lib/libcuda.so.1', '/usr/lib64/libcuda.so', '/usr/lib64/libcuda.so.1'])
>>> print(numba.cuda.cudadrv.driver.driver.cuInit(0))
Traceback (most recent call last):
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 292, in ensure_initialized
self.cuInit(0)
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 343, in safe_cuda_api_call
self._check_ctypes_error(fname, retcode)
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 414, in _check_ctypes_error
raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [100] Call to cuInit results in CUDA_ERROR_NO_DEVICE
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 307, in __getattr__
self.ensure_initialized()
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 296, in ensure_initialized
raise CudaSupportError(f"Error at driver init: {description}")
numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: Call to cuInit results in CUDA_ERROR_NO_DEVICE (100)
- After the change of the statements order, the warning message in the above code disappears. However, when there are cross-file call (both files have
import cudf.pandas), an error is returned.
(cuda) me@deskwin10:~/code$ python nn_min.py > tmp1.txt
Traceback (most recent call last):
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 292, in ensure_initialized
self.cuInit(0)
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 343, in safe_cuda_api_call self._check_ctypes_error(fname, retcode) File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 414, in _check_ctypes_error
raise CudaAPIError(retcode, msg) numba.cuda.cudadrv.driver.CudaAPIError: [100] Call to cuInit results in CUDA_ERROR_NO_DEVICE
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/fast_slow_proxy.py", line 997, in _fast_slow_function_call result = func(*fast_args, **fast_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/fast_slow_proxy.py", line 28, in call_operator
return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/utils/performance_tracking.py", line 51, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/core/frame.py", line 1977, in __str__ return repr(self) ^^^^^^^^^^ File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/utils/performance_tracking.py", line 51, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/core/dataframe.py", line 2161, in __repr__ return self._clean_renderable_dataframe(output) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/core/dataframe.py", line 2060, in _clean_renderable_dataframe output = output.to_pandas().to_string( ^^^^^^^^^^^^^^^^^^ File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/utils/performance_tracking.py", line 51, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/core/dataframe.py", line 5675, in to_pandas i: col.to_pandas(nullable=nullable, arrow_type=arrow_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/core/column/numerical.py", line 696, in to_pandas return pd.Index(self.values_host, copy=False)
^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/core/column/column.py", line 724, in values_host
return self.data_array_view(mode="read").copy_to_host()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/core/column/column.py", line 628, in data_array_view
return cuda.as_cuda_array(obj).view(self.dtype)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/api.py", line 77, in as_cuda_array
return from_cuda_array_interface(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/devices.py", line 236, in _require_cuda_context
with _runtime.ensure_context():
File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/devices.py", line 123, in ensure_context
with driver.get_active_context():
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 512, in __enter__
driver.cuCtxGetCurrent(byref(hctx))
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 307, in __getattr__
self.ensure_initialized()
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 296, in ensure_initialized
raise CudaSupportError(f"Error at driver init: {description}")
numba.cuda.cudadrv.error.CudaSupportError: Error at driver init: Call to cuInit results in CUDA_ERROR_NO_DEVICE (100)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): 19:42:33 [34/946]
File "/home/me/code/nn_min.py", line 266, in <module>
main()
File "/home/me/code/nn_min.py", line 243, in main
X, y, time_stamp, flag_sparse = ts_feature_creation(df, rw, pw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/me/code/feature_creation.py", line 273, in ts_feature_creation
print(df.loc[df.isnull().any(axis=1),df.columns])
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/fast_slow_proxy.py", line 722, in __call__
result, _ = _fast_slow_function_call(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/fast_slow_proxy.py", line 1040, in _fast_slow_function_call
slow_args, slow_kwargs = _slow_arg(args), _slow_arg(kwargs)
^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/fast_slow_proxy.py", line 1191, in _slow_arg
return _transform_arg(arg, "_fsproxy_slow", seen)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/fast_slow_proxy.py", line 1108, in _transform_arg
return tuple(
^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/fast_slow_proxy.py", line 1109, in <genexpr>
_transform_arg(a, attribute_name, seen) for a in arg
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/fast_slow_proxy.py", line 1108, in _transform_arg
return tuple(
^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/fast_slow_proxy.py", line 1109, in <genexpr>
_transform_arg(a, attribute_name, seen) for a in arg
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/fast_slow_proxy.py", line 1064, in _transform_arg
typ = getattr(arg, attribute_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/fast_slow_proxy.py", line 540, in _fsproxy_slow
self._fsproxy_wrapped = self._fsproxy_fast_to_slow()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/fast_slow_proxy.py", line 204, in _fsproxy_fast_to_slow
return fast_to_slow(self._fsproxy_wrapped)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/pandas/_wrappers/pandas.py", line 248, in <lambda>
fast_to_slow=lambda fast: fast.to_pandas(),
^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/utils/performance_tracking.py", line 51, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/core/dataframe.py", line 5675, in to_pandas
i: col.to_pandas(nullable=nullable, arrow_type=arrow_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/core/column/numerical.py", line 696, in to_pandas
return pd.Index(self.values_host, copy=False)
^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/core/column/column.py", line 724, in values_host
return self.data_array_view(mode="read").copy_to_host()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/cudf/core/column/column.py", line 628, in data_array_view
return cuda.as_cuda_array(obj).view(self.dtype)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/api.py", line 77, in as_cuda_array
return from_cuda_array_interface(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/devices.py", line 236, in _require_cuda_context
with _runtime.ensure_context():
File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/devices.py", line 123, in ensure_context
with driver.get_active_context():
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 512, in __enter__
driver.cuCtxGetCurrent(byref(hctx))
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/share/.virtualenvs/torch_cuda/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py", line 310, in __getattr__
raise CudaSupportError(
numba.cuda.cudadrv.error.CudaSupportError: Error at driver init:
Call to cuInit results in CUDA_ERROR_NO_DEVICE (100):
A few other notes that may be helpful in debugging:
Can you share more about how you installed the CUDA Toolkit? On GeForce cards, forward compatibility is not supported. That means that your driver must be at least as new as the CUDA Toolkit version you use. If you're using CUDA Toolkit 12.9 (the latest), you also need to update your driver to the latest. I see you have driver 560.94 and the latest version for Windows with RTX 2060 is 576.52.
On WSL, the driver you get should be from the host Windows system -- you do not need a driver in the Linux OS. If you have a driver installed in your WSL instance, uninstall it first!
On WSL, you will need the
nvidia-container-toolkitto be installed if you run in a Docker container. You can follow the instructions here: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.htmlThe short list of commands for that is:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.listsudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
Don't forget to set up Docker!
sudo nvidia-ctk runtime configure --runtime=docker
I downloaded the driver from the official website via Manual Driver Search. Updating to driver 576.52 does not help. I don't have any other container except for the WSL2 in WIN 10.