TensorRT-LLM

TensorRT-LLM copied to clipboard

Llama 2 Execution Bug

4

### System Info CPU: x86_64, memory: 1024GB, GPU: 8*A6000 48GB each, Tensorrt-LLM version 0.9.0.DEV20240226. NVIDIA-Driver Version: 535.171.04 CUDA Version: 12.2; OS - Ubuntu 22.04 ### Who can help? _No response_...

Hudayday

ammo_cuda_ext and ammo_cuda_ext_fp8 building failed

1

### System Info CPU: Intel(R) Xeon(R) Platinum 8369B, GPU: a single NVIDIA A10, Driver Version: 550.54.14, CUDA Version: 12.4, NVCC Version: 12.1.105, TensorRT-LLM Version: 0.9.0.dev2024022700, nvidia-ammo Version: 0.7.4 ### Who...

Phoveran

zhangxin81

Is there any feature related to GPT-like models that can be applied to BERT-like models?

3

Is there any fesature related to GPT-like models that can be applied to BERT-like models?

zhangxin81

Missing kernels for sm_87 (Jetson Orin AGX)

1

### System Info Jetson Orin AGX, using the version 0.10 from pip ### Who can help? _No response_ ### Information - [X] The official example scripts - [ ] My...

hidoba

Llama 3 build error

1

Trying to build an engine for llama 3 70b. I get `KeyError : "Architecture"`

nivibilla

Segmentation fault when reproducing Throughput Benchmark (gptj and llama are both problematic)

1

### System Info NVIDIA A100-SXM4-80GB ### Who can help? _No response_ ### Information - [ ] The official example scripts - [ ] My own modified scripts ### Tasks -...

shixuan94

### System Info A100 GPUs (40GB) ### Who can help? @byshiue ### Information - [ ] The official example scripts - [ ] My own modified scripts ### Tasks -...

vnkc1

Parameter --load_model_on_cpu is ignored by llama convert_checkpoint.py (version 0.9.0)

1

### System Info - OS: Ubuntu 22.04.4 LTS - Nvidia driver version: 545.23.08 - CPU architecture: x86 - RAM size: ~500GB - GPUs: 2xL40s 48GB - Docker container image: manually...

fedem96

Question about the relation between ` --max_input_len --max_output_len` at building time and `input_output_len` at benchmark .

1

### System Info NVIDIA H20 97871MiB * 8 trt-llm 0.9.0 ### Who can help? _No response_ ### Information - [X] The official example scripts - [ ] My own modified...

YiandLi