Crash with Cuda: [ONNXRuntimeError] INVALID_ARGUMENT : unsupported conv activation mode "LeakyRelu"

[Shinmera]

Hi, thank you very much for providing such an easy to use and integrate project for face tracking!

I was trying to get the CPU usage down, since even on my pretty modern system it maxes things out hard (more on that later). So I installed the Arch Linux package variants python-onnxruntime-opt and python-opencv-cuda in the hopes that it would help reduce the load. However, trying to run facetracker.py now results in the following crash:

> python facetracker.py
[ WARN:[email protected]] global cap_gstreamer.cpp:1777 open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
[ WARN:[email protected]] global cap_gstreamer.cpp:2152 setProperty OpenCV | GStreamer warning: GStreamer: unhandled property
2025-06-15 19:23:49.101741927 [E:onnxruntime:, inference_session.cc:2280 operator()] Exception during initialization: /usr/src/debug/onnxruntime/onnxruntime-opt-cuda/onnxruntime/contrib_ops/cuda/fused_conv.cc:68 onnxruntime::contrib::cuda::FusedConv<T>::FusedConv(const onnxruntime::OpKernelInfo&) [with T = float] [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : unsupported conv activation mode "LeakyRelu"
Stacktrace:
/usr/lib/python3.13/site-packages/onnxruntime/capi/libonnxruntime_providers_cuda.so(+0x464c999d) [0x7f45dc6c999d]
/usr/lib/python3.13/site-packages/onnxruntime/capi/libonnxruntime_providers_cuda.so(+0x4616428b) [0x7f45dc36428b]
/usr/lib/python3.13/site-packages/onnxruntime/capi/libonnxruntime_providers_cuda.so(+0x46537fa1) [0x7f45dc737fa1]
/usr/lib/python3.13/site-packages/onnxruntime/capi/libonnxruntime_providers_cuda.so(+0x461e6fdb) [0x7f45dc3e6fdb]
/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0xfc26c0) [0x7f46029c26c0]
/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0xb788be) [0x7f46025788be]
/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0x46c8d5) [0x7f4601e6c8d5]
/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0x390a8e) [0x7f4601d90a8e]
/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0x3a4be4) [0x7f4601da4be4]
/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0x35bd51) [0x7f4601d5bd51]
/usr/lib/libpython3.13.so.1.0(+0x191fcc) [0x7f4754391fcc]
/usr/lib/libpython3.13.so.1.0(_PyObject_MakeTpCall+0x2cb) [0x7f475435d46b]
/usr/lib/libpython3.13.so.1.0(_PyEval_EvalFrameDefault+0x1344) [0x7f475436ecd4]
/usr/lib/libpython3.13.so.1.0(+0x1a5a40) [0x7f47543a5a40]
/usr/lib/libpython3.13.so.1.0(_PyObject_MakeTpCall+0x21c) [0x7f475435d3bc]
/usr/lib/libpython3.13.so.1.0(_PyEval_EvalFrameDefault+0xadf8) [0x7f4754378788]
/usr/lib/libpython3.13.so.1.0(+0x1a5a40) [0x7f47543a5a40]
/usr/lib/libpython3.13.so.1.0(_PyObject_MakeTpCall+0x21c) [0x7f475435d3bc]
/usr/lib/libpython3.13.so.1.0(_PyEval_EvalFrameDefault+0xadf8) [0x7f4754378788]
/usr/lib/libpython3.13.so.1.0(+0x1a5a40) [0x7f47543a5a40]
/usr/lib/libpython3.13.so.1.0(_PyObject_MakeTpCall+0x21c) [0x7f475435d3bc]
/usr/lib/libpython3.13.so.1.0(_PyEval_EvalFrameDefault+0xadf8) [0x7f4754378788]
/usr/lib/libpython3.13.so.1.0(PyEval_EvalCode+0xb5) [0x7f4754441695]
/usr/lib/libpython3.13.so.1.0(+0x27f433) [0x7f475447f433]
/usr/lib/libpython3.13.so.1.0(+0x27c81a) [0x7f475447c81a]
/usr/lib/libpython3.13.so.1.0(+0x279f27) [0x7f4754479f27]
/usr/lib/libpython3.13.so.1.0(+0x2791e0) [0x7f47544791e0]
/usr/lib/libpython3.13.so.1.0(+0x278ff3) [0x7f4754478ff3]
/usr/lib/libpython3.13.so.1.0(Py_RunMain+0x3f4) [0x7f4754477244]
/usr/lib/libpython3.13.so.1.0(Py_BytesMain+0x3c) [0x7f475442e95c]
/usr/lib/libc.so.6(+0x276b5) [0x7f47540376b5]
/usr/lib/libc.so.6(__libc_start_main+0x89) [0x7f4754037769]
python(_start+0x25) [0x55bdd2550045]

Traceback (most recent call last):
  File "/run/media/data/Projects/cl/vtryout/OpenSeeFace/facetracker.py", line 250, in <module>
    tracker = Tracker(width, height, threshold=args.threshold, max_threads=args.max_threads, max_faces=args.faces, discard_after=args.discard_after, scan_every=args.scan_every, silent=False if args.silent == 0 else True, model_type=args.model, model_dir=args.model_dir, no_gaze=False if args.gaze_tracking != 0 and args.model != -1 else True, detection_threshold=args.detection_threshold, use_retinaface=args.scan_retinaface, max_feature_updates=args.max_feature_updates, static_model=True if args.no_3d_adapt == 1 else False, try_hard=args.try_hard == 1)
  File "/run/media/data/Projects/cl/vtryout/OpenSeeFace/tracker.py", line 526, in __init__
    self.retinaface = RetinaFaceDetector(model_path=os.path.join(model_base_path, "retinaface_640x640_opt.onnx"), json_path=os.path.join(model_base_path, "priorbox_640x640.json"), threads=max(max_threads,4), top_k=max_faces, res=(640, 640))
                      ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/run/media/data/Projects/cl/vtryout/OpenSeeFace/retinaface.py", line 70, in __init__
    self.session = onnxruntime.InferenceSession(model_path, sess_options=options, providers=providersList)
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 472, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 561, in _create_inference_session
    sess.initialize_session(providers, provider_options, disabled_optimizers)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /usr/src/debug/onnxruntime/onnxruntime-opt-cuda/onnxruntime/contrib_ops/cuda/fused_conv.cc:68 onnxruntime::contrib::cuda::FusedConv<T>::FusedConv(const onnxruntime::OpKernelInfo&) [with T = float] [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : unsupported conv activation mode "LeakyRelu"
Stacktrace:
/usr/lib/python3.13/site-packages/onnxruntime/capi/libonnxruntime_providers_cuda.so(+0x464c999d) [0x7f45dc6c999d]
/usr/lib/python3.13/site-packages/onnxruntime/capi/libonnxruntime_providers_cuda.so(+0x4616428b) [0x7f45dc36428b]
/usr/lib/python3.13/site-packages/onnxruntime/capi/libonnxruntime_providers_cuda.so(+0x46537fa1) [0x7f45dc737fa1]
/usr/lib/python3.13/site-packages/onnxruntime/capi/libonnxruntime_providers_cuda.so(+0x461e6fdb) [0x7f45dc3e6fdb]
/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0xfc26c0) [0x7f46029c26c0]
/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0xb788be) [0x7f46025788be]
/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0x46c8d5) [0x7f4601e6c8d5]
/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0x390a8e) [0x7f4601d90a8e]
/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0x3a4be4) [0x7f4601da4be4]
/usr/lib/python3.13/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so(+0x35bd51) [0x7f4601d5bd51]
/usr/lib/libpython3.13.so.1.0(+0x191fcc) [0x7f4754391fcc]
/usr/lib/libpython3.13.so.1.0(_PyObject_MakeTpCall+0x2cb) [0x7f475435d46b]
/usr/lib/libpython3.13.so.1.0(_PyEval_EvalFrameDefault+0x1344) [0x7f475436ecd4]
/usr/lib/libpython3.13.so.1.0(+0x1a5a40) [0x7f47543a5a40]
/usr/lib/libpython3.13.so.1.0(_PyObject_MakeTpCall+0x21c) [0x7f475435d3bc]
/usr/lib/libpython3.13.so.1.0(_PyEval_EvalFrameDefault+0xadf8) [0x7f4754378788]
/usr/lib/libpython3.13.so.1.0(+0x1a5a40) [0x7f47543a5a40]
/usr/lib/libpython3.13.so.1.0(_PyObject_MakeTpCall+0x21c) [0x7f475435d3bc]
/usr/lib/libpython3.13.so.1.0(_PyEval_EvalFrameDefault+0xadf8) [0x7f4754378788]
/usr/lib/libpython3.13.so.1.0(+0x1a5a40) [0x7f47543a5a40]
/usr/lib/libpython3.13.so.1.0(_PyObject_MakeTpCall+0x21c) [0x7f475435d3bc]
/usr/lib/libpython3.13.so.1.0(_PyEval_EvalFrameDefault+0xadf8) [0x7f4754378788]
/usr/lib/libpython3.13.so.1.0(PyEval_EvalCode+0xb5) [0x7f4754441695]
/usr/lib/libpython3.13.so.1.0(+0x27f433) [0x7f475447f433]
/usr/lib/libpython3.13.so.1.0(+0x27c81a) [0x7f475447c81a]
/usr/lib/libpython3.13.so.1.0(+0x279f27) [0x7f4754479f27]
/usr/lib/libpython3.13.so.1.0(+0x2791e0) [0x7f47544791e0]
/usr/lib/libpython3.13.so.1.0(+0x278ff3) [0x7f4754478ff3]
/usr/lib/libpython3.13.so.1.0(Py_RunMain+0x3f4) [0x7f4754477244]
/usr/lib/libpython3.13.so.1.0(Py_BytesMain+0x3c) [0x7f475442e95c]
/usr/lib/libc.so.6(+0x276b5) [0x7f47540376b5]
/usr/lib/libc.so.6(__libc_start_main+0x89) [0x7f4754037769]
python(_start+0x25) [0x55bdd2550045]

Relevant package versions are:

python-onnxruntime-opt 1.22.0-5 onnxruntime-opt 1.22.0-5 python-opencv-cuda 4.11.0-13 opencv-cuda 4.11.0-13 cudnn 9.10.2.21-1 cuda 12.9.1-1 nccl 2.27.3-1

I'm afraid this is all quite out of my depth, so I'm not sure what steps to take to fix this issue or help debug it further. If there's anything I can do, please let me know.

By the way: it also seems like I can't pass the -W and -H arguments when using a camera source on Linux, so I assume it might be using a ridiculously high resolution offered by my camera, leading to the aforementioned high CPU load. If there's a way to reduce that, too, that'd be great.