Katherine Yang

icon company NVIDIA

Results 100 comments of Katherine Yang

tutorials/Quick_Deploy/vLLM the triton metrics inference delay result is abnormal？

I meant that how did you get 300-500ms? But I agree, the metrics should behave the same way as it would have with Fastertransformer backend.

[Java] Add FP16 Support for Java Client

@krishung5 can you take a look? I can review sometimes next week. @du00cs can you send us the CLA? ref: https://github.com/triton-inference-server/server/blob/main/CONTRIBUTING.md

Server Hangs when loading python backend w/ pytorch (including the example)

Closing issue due to lack of activity. Please re-open the issue if you would like to follow up with this issue

example for Kubernetes configuration multi-node multi-gpu

ping @rmccorm4 ?

Tritonbackend multiple instances of the same model run sequentially instead of in parallel on the same device with asyncrhonous requests.

Closing issue due to lack of activity. Please re-open the issue if you would like to follow up with this issue

Possible GPU memory leak in Triton. Not draining.

Closing issue due to lack of activity. Please re-open the issue if you would like to follow up with this issue

Using shm in bls

Closing issue due to lack of activity. Please re-open the issue if you would like to follow up with this issue.

Get shm memory status

Closing issue due to lack of activity. Please re-open the issue if you would like to follow up with this issue

Shared memory failing in gunicorn following example

Closing issue due to lack of activity. Please re-open the issue if you would like to follow up with this issue

`tritonclient[grpc]==2.24.0` Produces OOMs When Async gRPC Calls Are Performed

Closing issue due to lack of activity. Please re-open the issue if you would like to follow up with this issue