tensorrtllm_backend

tensorrtllm_backend copied to clipboard

Failed to launch triton server, the tensorrt_llm protobuf file failed to load

2

### System Info Docker image: nvcr.io/nvidia/tritonserver:24.07-trtllm-python-py3 Device: 8x H100 trt-llm backend: v0.11.0 ### Who can help? @byshiue @schetlur-nv ### Information - [ ] The official example scripts - [X] My...

KuntaiDu

krishung5

LLAMA3: Unable to launch with tp 2

2

### System Info - GPU: H100 - Triton Server with Tensor rt Backend (v.0.10.0) - Launched on K8s. Docker Container built using [tensor rt builder](https://github.com/triton-inference-server/tensorrtllm_backend/tree/v0.10.0/dockerfile) - K8s Container uses Shared...

mindhash

T5 model: Encountered an error when fetching new request: Prompt length (200) exceeds maximum input length (1)

1

### System Info L4 GPU GPU memory: 24 GB TensorRT LLM version: v0.10.0 container used: tritonserver:24.06-trtllm-python-py3 ### Who can help? @byshiue @schetlur-nv ### Information - [X] The official example scripts...

jayakommuru

Build Qwen2-72B model to INT4-AWQ TensorRT engines failed

1

### System Info - Ubuntu 20.04 - NVIDIA A100 ### Who can help? @Tracin @kaiyux ### Information - [X] The official example scripts - [ ] My own modified scripts...

wangpeilin

v0.11.0 release fails when TP>1

1

### System Info - CPU: x86_64 - GPUs: 8x H100 80GB HBM3 - Driver: 550.90.07 - CUDA: 12.4 - TensorRT-LLM: v0.11.0 - tensorrtllm_backend: v0.11.0 ### Who can help? @kaiyux ###...

daulet

Triton crashes on boot

1

### System Info - Hardware: 8x NVIDIA H100 80GB HBM3 - Software: NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.4 - tensorrtllm_backend commit: [d173386f4dd7b3ed5883dea43851b6bde5eda5c7](https://github.com/triton-inference-server/tensorrtllm_backend/commit/d173386f4dd7b3ed5883dea43851b6bde5eda5c7) ### Who can help? _No response_...

daulet

### System Info triton images:24.07-trtllm-python-py3 ### Who can help? _No response_ ### Information - [ ] The official example scripts - [ ] My own modified scripts ### Tasks -...

gzy19990617

### System Info GPU Name: NVIDIA A800 TensorRT-LLM: 0.11.0 Nvidia Driver: 535.129.03 OS: Ubuntu 22.04 triton-inference-server backend：tensorrtllm_backend ### Who can help? _No response_ ### Information - [ ] The official...

limertang

### System Info GPU Name: NVIDIA A800 TensorRT-LLM: 0.11.0 Nvidia Driver: 535.129.03 OS: Ubuntu 22.04 triton-inference-server backend：tensorrtllm_backend ### Who can help? _No response_ ### Information - [ ] The official...

limertang