lss15151161

[Bug] llava-v1.6-34b can not enable Tensor Parallelism, server can not start

1


for llava， how to send image rather than image_path?

2

current example(srt_example_llava.py) request like this: state = image_qa.run( image_path="./images/cat.jpeg", question="What is this?", max_new_tokens=64 ) but, if i want to send request like below, what should I do? request = {...

The llava model batch inference result is different with batch=1

5

### System info GPU: A100 tensorrt 9.3.0.post12.dev1 tensorrt-llm 0.9.0 torch 2.2.2 ### Reproduction ``` export MODEL_NAME="llava-1.5-7b-hf" git clone https://huggingface.co/llava-hf/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} ``` ``` python ../llama/convert_checkpoint.py \ --model_dir tmp/hf_models/${MODEL_NAME} \ --output_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu...

llava batch infer, only the result corresponding to the longest prompt is correct, while other results are incorrect

3

version: TensorRT-LLM 0.10.0 the official script(TensorRT-LLM/examples/multimodal/run.py) use same prompt repeat to form a batch. but if I use different prompts to form a batch, the result is incorrect. how to...