MiniCPM-o
MiniCPM-o copied to clipboard
OpenBMB
[BUG] vllm 跑openbmb/MiniCPM-o-2_6-int4
是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?
- [x] 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions
该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?
- [x] 我已经搜索过FAQ | I have searched FAQ
当前行为 | Current Behavior
我用vllm 跑openbmb/MiniCPM-o-2_6-int4,代码参考的官方的DEMO部分，卡用的是RTX 2080TI，为什么会报代码片段和出错片段如下，代码片段只是改了模型名字如下： from transformers import AutoTokenizer from PIL import Image from vllm import LLM, SamplingParams
MODEL_NAME = "openbmb/MiniCPM-o-2_6-int4"
MODEL_NAME = "openbmb/MiniCPM-O-2_6"
Also available for previous models
MODEL_NAME = "openbmb/MiniCPM-Llama3-V-2_5"
MODEL_NAME = "HwwwH/MiniCPM-V-2"
image = Image.open("xxx.png").convert("RGB") tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True) llm = LLM( model=MODEL_NAME, trust_remote_code=True, gpu_memory_utilization=1, max_model_len=2048, dtype='half'
)
messages = [{
"role":
"user",
"content":
# Number of images
"(
"\nWhat is the content of this image?"
}]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
Single Inference
inputs = {
"prompt": prompt,
"multi_modal_data": {
"image": image
# Multi images, the number of images should be equal to that of (<image>./</image>)
# "image": [image, image]
},
}
Batch Inference
inputs = [{。。。。。其余不变
出错片段：
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/openbmb-vllm/./tests.py", line 13, in
期望行为 | Expected Behavior
No response
复现方法 | Steps To Reproduce
No response
运行环境 | Environment
- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA (`python -c 'import torch; print(torch.version.cuda)'`):
备注 | Anything else?
No response
