Kris Hung comments

Results 129 comments of


                                            Kris Hung

build error: The batch manager library is truncated or incomplete

I remember I saw something similar when TRT was not correctly installed or the path was not set. The `main` includes some new changes related to the build, which is...

Triton's gpumemory footprint is twice that of Tensorrt-LLM

@lyc728 I was wondering what's the build failure that you see in 23.12. Also, could you provide us the steps to reproduce the GPU memory footprint so that we can...

`pb_utils.TritonError.NOT_HEALTHY` Error Code

I think that this request will help users to better handle the model health issues. Filed a feature request ticket (DLIS-6039).

why can not cancel the request in the first model of ensemble_model

@kthui please correct me if I'm wrong. From the [documentation](https://github.com/triton-inference-server/server/blob/r23.10/docs/user_guide/request_cancellation.md#handling-in-triton-core) regarding how Triton core handles cancelled requests, I think currently the check only happens between each step in ensemble models....

SEVERE: Inference failed on 3th retry

@yangkl96 Sorry for the late reply. Could you provide the commands/minimum steps to reproduce the error? There seems to be some connection issue, I was wondering if you are running...

TritonModelException: Failed to open the cudaIpcHandle when I call the pytorch model upon wsl2 ubuntu 22.04

Hi @youwan114, can you share the output of `nvidia-smi` to check the CUDA driver version? I remember we have seen similar issue before, and upgrading the CUDA driver version helped.

TritonModelException: Failed to open the cudaIpcHandle when I call the pytorch model upon wsl2 ubuntu 22.04

Thanks for providing the logs. Can you share the `docker run ...` command that you use for running the container? I wonder if adding `--pid host` flag to the command...

TritonModelException: Failed to open the cudaIpcHandle when I call the pytorch model upon wsl2 ubuntu 22.04

Apologize for the delay reply. Regarding > I0824 05:50:11.703413 5921 http_server.cc:1103] HTTP: unable to provide 'output__0' in GPU, will use CPU I0824 05:50:11.703528 5921 http_server.cc:1103] HTTP: unable to provide 'output__1'...

The Docker build process has failed.

Hi @is, I wasn't able to reproduce the build issue. Could you please share the command you used to run the docker container? I'm wondering if there are any HTTP...

[Performance] Regression observed when using CUDA execution provider

@gedoensmax I am using cuDNN 8.7.0.84. Tried to use cuDNN 9 with onnxruntime-gpu 1.17.1 but it's finding cuDNN 8. ``` 2024-05-17 19:57:13.917623151 [E:onnxruntime:Default, provider_bridge_ort.cc:1548 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] :...