talmolab
CUDA kernal binaries not compatible with GPU
Bug description
My new GPU doesn't seem to be supported on the current version of CUDA that comes with the SLEAP environment.
Expected behaviour
tracking runs quickly on the video file
Actual behaviour
SLEAP has to JIT compile the GPU kernels which takes like 10 mins per video and adds a ton of overhead time I get this warning message:
"TensorFlow was not built with CUDA kernel binaries compatible with compute capability 12.0. CUDA kernels will be jit-compiled from PTX, which could take 30 minutes or longer."
Your personal set up
GPU: NVIDIA 5090RTX System: intel core ultra 9
- OS: intel core ultra 9
- Version(s):
- SLEAP installation method (listed here):
- [X] Conda from package
- [ ] Conda from source
- [ ] pip package
- [ ] Apple Silicon Macs
Environment packages
# paste output of `pip freeze` or `conda list` here
Name Version Build Channel
absl-py 2.1.0 pypi_0 pypi albumentations 1.3.1 pyhd8ed1ab_0 conda-forge astunparse 1.6.3 pypi_0 pypi attrs 21.4.0 pyhd8ed1ab_0 conda-forge backports-zoneinfo 0.2.1 pypi_0 pypi brotli 1.1.0 h2466b09_2 conda-forge brotli-bin 1.1.0 h2466b09_2 conda-forge ca-certificates 2025.1.31 h56e8100_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.3.3 pypi_0 pypi cattrs 1.1.1 pyhd8ed1ab_0 conda-forge certifi 2024.8.30 pyhd8ed1ab_0 conda-forge charset-normalizer 3.3.2 pypi_0 pypi cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge cuda-nvcc 11.3.58 hb8d16a4_0 nvidia cudatoolkit 11.3.1 hf2f0253_13 conda-forge cudnn 8.2.1.32 h754d62a_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge cytoolz 0.12.0 py37hcc03f2d_0 conda-forge dask-core 2022.2.0 pyhd8ed1ab_0 conda-forge efficientnet 1.0.0 pypi_0 pypi flatbuffers 2.0.7 pypi_0 pypi fonttools 4.38.0 py37h51bd9d9_0 conda-forge freeglut 3.2.2 he0c23c2_3 conda-forge freetype 2.13.3 h0b5ce68_0 conda-forge fsspec 2023.1.0 pyhd8ed1ab_0 conda-forge gast 0.4.0 pypi_0 pypi google-auth 2.29.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.62.1 pypi_0 pypi h5py 3.7.0 nompi_py37h24adfc3_101 conda-forge hdf5 1.12.2 nompi_h2a0e4a3_101 conda-forge hdmf 3.6.1 pypi_0 pypi icu 69.1 h0e60522_0 conda-forge idna 3.6 pypi_0 pypi image-classifiers 1.0.0 pypi_0 pypi imagecodecs-lite 2019.12.3 py37h0b711f8_5 conda-forge imageio 2.36.0 pyh12aca89_1 conda-forge imgstore 0.2.9 pypi_0 pypi importlib-metadata 4.2.0 pypi_0 pypi importlib_resources 6.0.0 pyhd8ed1ab_0 conda-forge intel-openmp 2024.2.1 h57928b3_1083 conda-forge jasper 2.0.33 hc2e4405_1 conda-forge joblib 1.3.2 pyhd8ed1ab_0 conda-forge jpeg 9e hcfcfb64_3 conda-forge jsmin 3.0.1 pyhd8ed1ab_0 conda-forge jsonpickle 1.2 py_0 conda-forge jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge keras 2.7.0 pypi_0 pypi keras-applications 1.0.8 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.4.4 py37h8c56517_0 conda-forge krb5 1.20.1 h6609f42_0 conda-forge lcms2 2.14 h90d422f_0 conda-forge lerc 4.0.0 h63175ca_0 conda-forge libaec 1.1.3 h63175ca_0 conda-forge libblas 3.9.0 31_h641d27c_mkl conda-forge libbrotlicommon 1.1.0 h2466b09_2 conda-forge libbrotlidec 1.1.0 h2466b09_2 conda-forge libbrotlienc 1.1.0 h2466b09_2 conda-forge libcblas 3.9.0 31_h5e41251_mkl conda-forge libclang 18.1.1 pypi_0 pypi libcurl 8.1.2 h68f0423_0 conda-forge libdeflate 1.14 hcfcfb64_0 conda-forge libhwloc 2.11.2 default_hc8275d1_1000 conda-forge libiconv 1.18 h135ad9c_1 conda-forge liblapack 3.9.0 31_h1aa476e_mkl conda-forge liblapacke 3.9.0 31_h845c4fa_mkl conda-forge liblzma 5.8.1 h2466b09_0 conda-forge liblzma-devel 5.8.1 h2466b09_0 conda-forge libopencv 4.5.5 py37h542666b_10 conda-forge libpng 1.6.47 had7236b_0 conda-forge libprotobuf 3.20.3 h12be248_0 conda-forge libsodium 1.0.18 h8d14728_1 conda-forge libsqlite 3.49.1 h67fdade_2 conda-forge libssh2 1.10.0 h680486a_3 conda-forge libtiff 4.4.0 hc4f729c_5 conda-forge libwebp-base 1.5.0 h3b0e114_0 conda-forge libxcb 1.13 hcd874cb_1004 conda-forge libxml2 2.13.7 h442d1da_1 conda-forge libxslt 1.1.39 h3df6e99_0 conda-forge libzlib 1.3.1 h2466b09_2 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge markdown 3.4.4 pypi_0 pypi markdown-it-py 2.2.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.5 pypi_0 pypi matplotlib-base 3.5.3 py37hbaab90a_2 conda-forge mdurl 0.1.2 pyhd8ed1ab_0 conda-forge mkl 2024.2.2 h66d3029_15 conda-forge msys2-conda-epoch 20160418 1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge ndx-pose 0.1.1 pyhd8ed1ab_0 conda-forge networkx 2.7 pyhd8ed1ab_0 conda-forge nixio 1.5.3 pypi_0 pypi numpy 1.21.6 py37h2830a78_0 conda-forge oauthlib 3.2.2 pypi_0 pypi opencv 4.5.5 py37h03978a9_10 conda-forge openjpeg 2.5.0 hc9384bd_1 conda-forge openssl 1.1.1w hcfcfb64_0 conda-forge opt-einsum 3.3.0 pypi_0 pypi packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 1.3.5 py37h9386db6_0 conda-forge partd 1.4.1 pyhd8ed1ab_0 conda-forge patsy 0.5.6 pyhd8ed1ab_0 conda-forge pillow 9.2.0 py37h42a8222_2 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_1 conda-forge protobuf 3.20.3 pypi_0 pypi psutil 5.9.3 py37h51bd9d9_0 conda-forge pthread-stubs 0.4 hcd874cb_1001 conda-forge pthreads-win32 2.9.1 h2466b09_4 conda-forge py-opencv 4.5.5 py37h90c5f73_10 conda-forge pyasn1 0.5.1 pypi_0 pypi pyasn1-modules 0.3.0 pypi_0 pypi pygments 2.17.2 pyhd8ed1ab_0 conda-forge pykalman 0.9.7 pyhd8ed1ab_0 conda-forge pynwb 2.3.3 pypi_0 pypi pyparsing 3.1.4 pyhd8ed1ab_0 conda-forge pyrsistent 0.18.1 py37hcc03f2d_1 conda-forge pyside2 5.13.2 py37h760f651_8 conda-forge python 3.7.12 h7840368_100_cpython conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python-rapidjson 1.9 py37h7f67f24_0 conda-forge python_abi 3.7 4_cp37m conda-forge pytz 2024.2 pyhd8ed1ab_0 conda-forge pywavelets 1.3.0 py37h3a130e4_1 conda-forge pyyaml 6.0 py37hcc03f2d_4 conda-forge pyzmq 24.0.1 py37h7347f05_0 conda-forge qimage2ndarray 1.10.0 pypi_0 pypi qt 5.12.9 h556501e_6 conda-forge qtpy 2.4.2 pyhdecd6ff_0 conda-forge qudida 0.0.4 pyhd8ed1ab_0 conda-forge requests 2.31.0 pypi_0 pypi requests-oauthlib 2.0.0 pypi_0 pypi rich 13.8.1 pyhd8ed1ab_0 conda-forge ruamel.yaml 0.17.21 py37hcc03f2d_1 conda-forge ruamel.yaml.clib 0.2.6 py37hcc03f2d_1 conda-forge scikit-image 0.19.2 py37h9386db6_0 conda-forge scikit-learn 1.0 py37ha78be43_1 conda-forge scikit-video 1.1.11 pyh24bf2e0_0 conda-forge scipy 1.7.3 py37hb6553fb_0 conda-forge seaborn 0.12.2 hd8ed1ab_0 conda-forge seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge segmentation-models 1.0.1 pypi_0 pypi setuptools 59.8.0 py37h03978a9_1 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sleap 1.4.1a2 pypi_0 pypi sqlite 3.49.1 h2466b09_2 conda-forge statsmodels 0.13.2 py37h3a130e4_0 conda-forge tbb 2021.13.0 h62715c5_1 conda-forge tensorboard 2.11.2 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorflow 2.7.0 pypi_0 pypi tensorflow-estimator 2.7.0 pypi_0 pypi tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge tensorflow-io-gcs-filesystem 0.31.0 pypi_0 pypi termcolor 2.3.0 pypi_0 pypi threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tifffile 2020.6.3 py_0 conda-forge tk 8.6.13 h5226925_1 conda-forge toolz 0.12.1 pyhd8ed1ab_0 conda-forge typing-extensions 4.7.1 hd8ed1ab_0 conda-forge typing_extensions 4.7.1 pyha770c72_0 conda-forge tzdata 2024.1 pypi_0 pypi tzlocal 5.1 pypi_0 pypi ucrt 10.0.22621.0 h57928b3_1 conda-forge unicodedata2 14.0.0 py37hcc03f2d_1 conda-forge urllib3 2.0.7 pypi_0 pypi vc 14.3 h2b53caa_26 conda-forge vc14_runtime 14.42.34438 hfd919c2_26 conda-forge vs2015_runtime 14.42.34438 h7142326_26 conda-forge werkzeug 2.2.3 pypi_0 pypi wheel 0.42.0 pyhd8ed1ab_0 conda-forge wrapt 1.16.0 pypi_0 pypi xorg-libxau 1.0.11 hcd874cb_0 conda-forge xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge xz 5.8.1 h208afaa_0 conda-forge xz-tools 5.8.1 h2466b09_0 conda-forge yaml 0.2.5 h8ffe710_2 conda-forge zeromq 4.3.4 h0e60522_1 conda-forge zipp 3.15.0 pyhd8ed1ab_0 conda-forge zstd 1.5.7 hbeecb71_2 conda-forge
Logs
# paste relevant logs here, if any
Screenshots
How to reproduce
- Go to '...'
- Click on '....'
- Scroll down to '....'
- See error
Thank you for pointing this out. We are working to update our backend, but unfortunately at this time we do not support this GPU. We are hoping to have a new release soon. Thank you for your patience.
