swift

swift copied to clipboard

Can I deploy SWIFT-trained models without using SWIFT inference?

2

Thanks for your work! Since SWIFT doesn't support deployment (API server) for multimodal models, can I train models using SWIFT and deploy to other frameworks such as LMDeploy? Thanks!

babla9

你好，我已按照说明将ColossalAI版本的grok-1下载到本地，并使用项目脚本训练。然而，我发现模型载入时间很长（50分钟左右），而我使用ColossalAI的载入脚本只需要20分钟左右。请问这是为什么？我应该如何调整以快速载入？脚本设置如下： nproc_per_node=8 PYTHONPATH=../../.. \ model_dir="" data_dir="" torchrun \ --nproc_per_node=$nproc_per_node \ --master_port 29500 \ llm_sft.py \ --model_type grok-1 \ --model_id_or_path $model_dir \ --sft_type lora \ --tuner_backend peft \ --dtype bf16 \...

Di-Zayn

目前支持多模态模型部署了吗

2

**Describe the bug** 目前使用以下脚本部署internvl-v15显示部署成功： CUDA_VISIBLE_DEVICES=0 swift deploy \ --model_type internvl-chat-v1_5 \ --model_id_or_path /home/weight/internvl-chat-v1-5/ \ --use_flash_attn true \ --dtype bf16 使用openai格式脚本调用，可以获取模型名称，然后就报错： ```python from openai import OpenAI import base64 import time client...

LRHstudy

qwen1.5-32b-chat 使用vllm推理很慢

3

已经运行了5分钟,一条数据也没有跑出来 

zhangfan-algo

请问这种应该怎么做？

sunzx8

swift export 指定 --tensor_parallel_size --gpu_memory_utilization 感觉不管用

6

swift export 对百川2进行awq量化 指定 --tensor_parallel_size --gpu_memory_utilization 感觉根本不管用，指定4张卡，只会用两张卡，然后就是爆显存

LIUKAI0815

peft加载qwen1half_72b_chat的lora模型报错

2

While copying the parameter named "base_model.model.model.layers.78.self_attn.v_proj.lora_B.default.weight", whose dimensions in the model are torch.Size([8192, 8]) and whose dimensions in the checkpoint are torch.Size([8192, 8]), an exception occurred : ('get_p2p_access:torch_npu/csrc/core/npu/NPUPeerToPeerAccess.cpp:80 NPU error,...

jhjiang10

TypeError: Subscripted generics cannot be used with class and instance checks

6

hi, I installed the required environment following the steps, and then directly ran the Lora fine-tuning script for llama3-8b-instruct, but the following error occurred: Traceback (most recent call last): File...

Hsu5918

**Describe the feature** Please describe the feature requested here(请在这里描述需求) RLAIF-V effectively reduce the hallucination of different MLLMs **Paste any useful information** Paste any useful information, including papers, github links, etc.(请在这里描述其他有用的信息，比如相关的论文地址，github链接等)...

choyakawa

怎么做 batch infer 来提高显卡利用率呢?

1

我使用 llava 72B 推理时，batch_size=1 显卡利用率不高，怎么可以使用大 batch 推理提高利用率呢？我没有使用 vllm，是直接使用的 swift 推理 https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/llava%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5.md

thesby