cuda-python icon indicating copy to clipboard operation
cuda-python copied to clipboard

Add CUDA version compatibility check

Open Andy-Jost opened this issue 2 months ago • 16 comments

Summary

Adds a version compatibility check that warns users when cuda-bindings was compiled against a newer CUDA major version than the installed driver supports.

Changes

cuda-bindings

  • Added check_cuda_version_compatibility() function in cuda/bindings/utils/_version_check.py
  • Compares compile-time CUDA_VERSION vs runtime cuDriverGetVersion()
  • Exported from cuda.bindings.utils
  • Added comprehensive unit tests in tests/test_version_check.py

cuda-core

  • Device.__new__ calls check_cuda_version_compatibility() after cuInit succeeds
  • Imports the function directly from cuda.bindings.utils

Rationale

When cuda-bindings is built against CUDA 13 headers but the user's driver only supports CUDA 12, many features will silently fail or behave unexpectedly. This check provides early, clear feedback:

UserWarning: cuda-bindings was built against CUDA 13.0, but the installed driver 
only supports CUDA 12.8. Some features may not work correctly. Consider updating 
your NVIDIA driver. Set CUDA_PYTHON_DISABLE_VERSION_CHECK=1 to suppress this warning.

Design

  • Provided by cuda-bindings: The version check implementation lives in cuda.bindings.utils since it checks cuda-bindings' compile-time version
  • Invoked by cuda-core: Called when Device first triggers CUDA initialization
  • Runs once: Uses a module-level flag to ensure the check runs only once per process
  • Non-blocking: Warning only, does not prevent operation
  • Suppressible: Set CUDA_PYTHON_DISABLE_VERSION_CHECK=1 to disable
  • Major version only: No warning for minor version differences (handled by graceful degradation per PR #1409)

Future Work

We could not find a suitable place to invoke the version check automatically within cuda-bindings itself (e.g., hooking into cuInit), so the check is currently triggered by cuda-core. This may be revisited in the future.

Test Coverage

7 tests in cuda-bindings covering:

  • No warning when driver is newer
  • No warning when same major version
  • Warning when compile major > driver major
  • Warning only issued once per process
  • Suppression via environment variable
  • Silent handling of driver errors
  • Silent handling of missing CUDA_VERSION attribute

Related Work

  • Complements PR #1409 which handles graceful degradation for minor version mismatches

Andy-Jost avatar Dec 18 '25 19:12 Andy-Jost

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

copy-pr-bot[bot] avatar Dec 18 '25 19:12 copy-pr-bot[bot]

/ok to test 7ce325c07b559e9a66472f5ba7a99fe980e17c09

Andy-Jost avatar Dec 18 '25 19:12 Andy-Jost

/ok to test 1962e351a233d65f2288fd6d16a43f0b0b75d6ad

Andy-Jost avatar Dec 18 '25 19:12 Andy-Jost

Doc Preview CI :---: |

:rocket: View preview at
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/
|
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-core/
|
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-bindings/
|
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-pathfinder/

|

Preview will be ready when the GitHub Pages deployment is complete.

github-actions[bot] avatar Dec 18 '25 19:12 github-actions[bot]

/ok to test 73e5a4379fad3392c24499e245a521c86c53db42

Andy-Jost avatar Dec 19 '25 00:12 Andy-Jost

/ok to test ddd92bd3ad9ef82f9d166897ec172dc0d42c850f

Andy-Jost avatar Dec 19 '25 00:12 Andy-Jost

/ok to test 62dfccab0047bb6421358d011398bc4c661cb5d7

Andy-Jost avatar Jan 07 '26 21:01 Andy-Jost

/ok to test b2083ed64147ce95739d141179466c582c63ee9a

Andy-Jost avatar Jan 08 '26 18:01 Andy-Jost

/ok to test fdb3a7e89

Andy-Jost avatar Jan 08 '26 18:01 Andy-Jost

/ok to test https://github.com/NVIDIA/cuda-python/commit/acadb3183278aca6201be0d1b733c2eb03b57dd5

Andy-Jost avatar Jan 09 '26 21:01 Andy-Jost

/ok to test 3a5c210cd89c018dda27e6e659a36802b17bc077

Andy-Jost avatar Jan 13 '26 19:01 Andy-Jost

/ok to test 424a113185c5163ec1c8a719bed392051c63cda5

Andy-Jost avatar Jan 14 '26 01:01 Andy-Jost

/ok to test 6071609504b9c8aca4dc8bef7764a27ca2911f2f

Andy-Jost avatar Jan 14 '26 01:01 Andy-Jost

/ok to test 96532b226

Andy-Jost avatar Jan 14 '26 17:01 Andy-Jost

/ok to test 973af9b7a

Andy-Jost avatar Jan 14 '26 17:01 Andy-Jost

/ok to test 7f23b08bc

Andy-Jost avatar Jan 14 '26 18:01 Andy-Jost

/ok to test a368e0f249b051ed43c4b099a327a7e107425e0b

Andy-Jost avatar Jan 23 '26 19:01 Andy-Jost

Doc Preview CI :---: Preview removed because the pull request was closed or merged.

github-actions[bot] avatar Jan 23 '26 23:01 github-actions[bot]