jan
jan copied to clipboard
epic: Support TensorRT-LLM on Windows as Inference Engine
https://www.digitaltrends.com/computing/microsoft-nvidia-tensorrt-llm-update-ignite-2023/
Tasks
- [ ] Step by step docs for Jan <> Windows TensorRT-LLM - 1 day
- [ ] Updated code in
triton-tensorrt-llm
extension - 1 day
Reference https://github.com/NVIDIA/trt-llm-rag-windows/blob/release/1.0/app.py#L43