ishandhanani
ishandhanani
Hi team! Was this ever added? I'm looking through the release notes but cannot find support for TRT-LLM
I think a good first step would be to have it generate the model repo for the trtllm backend only. In the future it would be great if we could...
Will this be merged in?
@RonanKMcGovern - I don't think triton has native support for an OAI compatible style endpoint yet. My current flow is converting a HF model via trt-llm and then feeding the...
You cooked on this one @rmccorm4
It seems like cute dsl has officially released 0.4.3 https://pypi.org/project/nvidia-cutlass-dsl/4.3.0/. Should we update this in the official deps and remove it from dockerfile?
This regression started in https://github.com/NVIDIA/nv-ingest/pull/154.
I also have not changed anything. When I'm on main ``` - YOLOX_GRPC_ENDPOINT=yolox:8001 - YOLOX_HTTP_ENDPOINT=http://yolox:8000/v1/infer - YOLOX_INFER_PROTOCOL=grpc ``` When I'm on the last working branch [c707a2b2bec26fcdde32a440820ecea62487f482](https://github.com/NVIDIA/nv-ingest/tree/c707a2b2bec26fcdde32a440820ecea62487f482) ``` - YOLOX_GRPC_ENDPOINT=yolox:8001 -...
@angudadevops - check out https://github.com/ishandhanani/nv-ingest. This is a working example. Steps to fix 1. Reverted main back to [c707a2b2bec26fcdde32a440820ecea62487f482](https://github.com/NVIDIA/nv-ingest/tree/c707a2b2bec26fcdde32a440820ecea62487f482) 2. For some reason - Uvicorn did not show up in...
@angudadevops - can you try to exec into the container? Check if uvicorn is installed