NeMo icon indicating copy to clipboard operation
NeMo copied to clipboard

Quartznet onnx model can't run on onnxruntime(CUDA backend) benchmark

Open wwiwush opened this issue 1 year ago • 1 comments

The develop ENV is as follows: cuda 11.8 cudnn 8.2.4 onnxruntime(cuda backend) 1.16.3

When I run my exported Quartznet onnx model on onnxruntime(CUDA backend) benchmark, the following error occurs: [E:onnxruntime:benchmark, cuda_call.cc:116 CudaCall] CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; file=/onnxruntime_src/onnxruntime/contrib_ops/cuda/fused_conv.cc ; line=85 ; expr=cudnnAddTensor(cudnnHandle, &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data) [E:onnxruntime:, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running FusedConv node. Name:'Conv_41' Status Message: CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ;file=/onnxruntime_src/onnxruntime/contrib_ops/cuda/fused_conv.cc ; line=85 ; expr=cudnnAddTensor(cudnnHandle, &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data); terminate called after throwing an instance of 'Ort::Exception' what(): Non-zero status code returned while running FusedConv node. Name:'Conv_41' Status Message: CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ;file=/onnxruntime_src/onnxruntime/contrib_ops/cuda/fused_conv.cc ; line=85 ; expr=cudnnAddTensor(cudnnHandle, &alpha, Base::s_.z_tensor, Base::s_.z_data, &alpha, Base::s_.y_tensor, Base::s_.y_data);

Thank you very much for help!

wwiwush avatar Jan 12 '24 09:01 wwiwush

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

github-actions[bot] avatar Feb 12 '24 01:02 github-actions[bot]

This issue was closed because it has been inactive for 7 days since being marked as stale.

github-actions[bot] avatar Feb 19 '24 01:02 github-actions[bot]