Manickavela

Hi @csukuangfj , @yuekaizhang Observed that currently only CUDA EP support is there and TensorRT EP support is not there for onnxruntime. is there ay active developments going on for...

I tried adding triggering onnxruntime's tensorrt ep for zipfromer but the model performance was very bad, debugging further with standalone onnxruntime in python for Encoder models, will update if I...

Hi @csukuangfj , Currently I am doing some experiments with Zipformer models, let me know if there are active developments going on from your end for TensorRT low precision support,...

Try adding libcudnn.so lib path to LD_PRELOAD variable, run this below CMD before running inference `export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/lib `

Facing the same issue in ubuntu docker image

Hi @csukuangfj, TensorRT has several parameters, and these will be only valid if TensorRT provider is chosen, so I need your suggestion on either of below 2. 1. Putting TRT...

yes, I will send the PR for configs separately in some time.

Current perf Trt Vs Cuda Tensorrt csrc/online-zipformer2-transducer-model.cc:RunEncoder:445 Encoder Duration : **1.930044** ms csrc/online-zipformer2-transducer-model.cc:RunEncoder:445 Encoder Duration : **0.034984** ms csrc/online-zipformer2-transducer-model.cc:RunEncoder:445 Encoder Duration : **0.034912** ms csrc/online-websocket-server-impl.cc:Run:256 Warm up completed : 3...

I can try to get for CPU numbers, but i don't have any high performance CPU, (in between someone can add support for dnnl ep 🙂) But here the focus...

Hi @csukuangfj https://github.com/k2-fsa/sherpa-onnx/pull/992 will create configs for execution provider all together and integrate it with sessions. let me know if you have any other thoughts still WIP