swift icon indicating copy to clipboard operation
swift copied to clipboard

目前支持多模态模型部署了吗

Open LRHstudy opened this issue 1 month ago • 2 comments

Describe the bug 目前使用以下脚本部署internvl-v15显示部署成功: CUDA_VISIBLE_DEVICES=0 swift deploy
--model_type internvl-chat-v1_5
--model_id_or_path /home/weight/internvl-chat-v1-5/
--use_flash_attn true
--dtype bf16

使用openai格式脚本调用,可以获取模型名称,然后就报错:

from openai import OpenAI
import base64
import time

client = OpenAI(
    api_key="EMPTY",
    base_url="http://127.0.0.1:8000/v1/",
)

model_name = client.models.list().data[0].id
print(model_name)
st = time.time()
print_stream = True
stream = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "请描述一下图像"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": 'https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg'
                    },
                },
            ],
        },

    ],
    model=model_name,
    stream=print_stream,
    max_tokens=512,
    top_p=0.2,
)

if not print_stream:
    print(stream)
else:
    for part in stream:
        print(part.choices[0].delta.content or "", end="", flush=True)
print("process_time:{}".format(round(time.time() - st, 3)))

服务端显示信息: INFO: 127.0.0.1:40132 - "GET /v1/models HTTP/1.1" 200 OK INFO: 127.0.0.1:40132 - "POST /v1/chat/completions HTTP/1.1" 422 Unprocessable Entity

Your hardware and system info Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息,如CUDA版本,系统,GPU型号和torch版本等) swift版本: ms-swift 2.1.0.dev0

Additional context Add any other context about the problem here(在这里补充其他信息)

LRHstudy avatar May 22 '24 09:05 LRHstudy