NVIDIA
Does Mistral can run on different GPUs(A100 & A5500) ?
I follow the instruction and can run Mistral with llama on GPU A100 or A5500 but I don't know how to set tp_size=2 / workers=2 to get the convert_checkpoint config I try 2 GPUs on 1 device or 2 device(with 1 GPU), both are failed If any suggestion, please let me know. Thanks for the reply.
You could try referring the guide of standard llama or other model. To run TP, you should add --tp_size 2 during converting the ckpt. If you still encounter error, please share your scripts and the full log by following the issue template.
System Info
- CPU architecture - x86_64
- GPU properties
- GPU name: NVIDIA A100 memory size: 40G
- GPU name: NVIDIA A5500 memory size: 24G
- Libraries
- TensorRT-LLM branch or tag: v0.10.0
- TensorRT-LLM commit: dev2024041600
- Container used: yes
- NVIDIA driver version: 550.54.15
- OS: Ubuntu 22.04
Who can help?
@byshiue
Reproduction
python3 convert_checkpoint.py --model_dir ./mistral --tp_size 2 trtllm-build --checkpoint_dir ./tllm_checkpoint --output_dir /tmp/fp16 --gemm_plugin float16 --max_input_len 32256 python3 ../run.py --max_output_len=50 --tokenizer_dir ./mistral/ --engine_dir=/tmp/fp16 --max_attention_window_size=4096
Expected behavior
run success
actual behavior
[TensorRT-LLM] TensorRT-LLM version: 0.10.0.dev2024041600
[TensorRT-LLM] TensorRT-LLM version: 0.10.0.dev2024041600
[TensorRT-LLM][INFO] Engine version 0.10.0.dev2024041600 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Engine version 0.10.0.dev2024041600 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter kv_cache_quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'num_medusa_heads' not found
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter quant_algo will not be set.
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null
[TensorRT-LLM][WARNING] Optional value for parameter kv_cache_quant_algo will not be set.
[TensorRT-LLM][WARNING] Optional value for parameter num_medusa_heads will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'max_draft_len' not found
[TensorRT-LLM][WARNING] Optional value for parameter max_draft_len will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'num_medusa_heads' not found
[TensorRT-LLM][WARNING] Optional value for parameter num_medusa_heads will not be set.
[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'max_draft_len' not found
[TensorRT-LLM][WARNING] Optional value for parameter max_draft_len will not be set.
[TensorRT-LLM][INFO] MPI size: 2, rank: 0
[TensorRT-LLM][INFO] MPI size: 2, rank: 1
Traceback (most recent call last):
File "/tmp/k8s/TensorRTLLM/examples/llama/../run.py", line 564, in
additional notes
none
Thank for the reply. Is there any way can run on different GPUs？ container recognize different GPUs as the same (ex. nvidia.com/gpu), is it possibe to run on different GPUs in container？
Is pp_size also? I set pp_size=2, it doesn't show any error message, but no more information
[TensorRT-LLM] TensorRT-LLM version: 0.10.0.dev2024041600 [TensorRT-LLM] TensorRT-LLM version: 0.10.0.dev2024041600 [TensorRT-LLM][INFO] Engine version 0.10.0.dev2024041600 found in the config file, assuming engine(s) built by new builder API. [TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null [TensorRT-LLM][WARNING] Optional value for parameter quant_algo will not be set. [TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null [TensorRT-LLM][WARNING] Optional value for parameter kv_cache_quant_algo will not be set. [TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'num_medusa_heads' not found [TensorRT-LLM][WARNING] Optional value for parameter num_medusa_heads will not be set. [TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'max_draft_len' not found [TensorRT-LLM][WARNING] Optional value for parameter max_draft_len will not be set. [TensorRT-LLM][INFO] MPI size: 2, rank: 0 [TensorRT-LLM][INFO] Engine version 0.10.0.dev2024041600 found in the config file, assuming engine(s) built by new builder API. [TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null [TensorRT-LLM][WARNING] Optional value for parameter quant_algo will not be set. [TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null [TensorRT-LLM][WARNING] Optional value for parameter kv_cache_quant_algo will not be set. [TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'num_medusa_heads' not found [TensorRT-LLM][WARNING] Optional value for parameter num_medusa_heads will not be set. [TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'max_draft_len' not found [TensorRT-LLM][WARNING] Optional value for parameter max_draft_len will not be set. [TensorRT-LLM][INFO] MPI size: 2, rank: 1 [TensorRT-LLM][INFO] Loaded engine size: 6923 MiB [TensorRT-LLM][INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 7082, GPU 7502 (MiB) [TensorRT-LLM][INFO] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 7083, GPU 7512 (MiB) [TensorRT-LLM][WARNING] TensorRT was linked against cuDNN 8.9.6 but loaded cuDNN 8.9.2 trt-deployment-6fc4ffd45-pb29j:1499:1499 [0] NCCL INFO Bootstrap : Using eth0:10.44.0.1<0> trt-deployment-6fc4ffd45-pb29j:1499:1499 [0] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v7 symbol. trt-deployment-6fc4ffd45-pb29j:1499:1499 [0] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v6 (v6) trt-deployment-6fc4ffd45-pb29j:1499:1499 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v7 symbol. trt-deployment-6fc4ffd45-pb29j:1499:1499 [0] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v6)
Traceback (most recent call last):
File "/tmp/k8s/TensorRTLLM/examples/llama/../run.py", line 564, in