lmdeploy
lmdeploy copied to clipboard
Published
1 month ago
•
InternLM
InternLM
[Bug] 使用lmdeploy部署internvl-38b-mpo-awq，在dify上使用接口的时候报错，是什么原因导致？
Checklist
- [ ] 1. I have searched related issues but cannot get the expected help.
- [ ] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
Describe the bug
请问什么原因导致？
Reproduction
模型启动命令： CUDA_VISIBLE_DEVICES=0,1 lmdeploy serve api_server /ssd/limingze_ssd/models/OpenGVLab/InternVL2_5-38B-MPO-AWQ --server-port 2556 --cache-max-entry-count 0.5 --tp 2 > internvl2.5_vl_38B_mpo_awq.log 2>&1 &
Environment
Package Version
---------------------------------------- -------------
accelerate 1.6.0
addict 2.4.0
aiohappyeyeballs 2.6.1
aiohttp 3.11.18
aiosignal 1.3.2
airportsdata 20250224
annotated-types 0.7.0
anyio 4.9.0
astor 0.8.1
attrs 25.3.0
av 14.3.0
blake3 1.0.4
cachetools 5.5.2
certifi 2025.4.26
charset-normalizer 3.4.2
click 8.1.8
cloudpickle 3.1.1
compressed-tensors 0.9.3
cupy-cuda12x 13.4.1
Deprecated 1.2.18
depyf 0.18.0
dill 0.4.0
diskcache 5.6.3
distro 1.9.0
dnspython 2.7.0
einops 0.8.1
email_validator 2.2.0
fastapi 0.115.12
fastapi-cli 0.0.7
fastrlock 0.8.3
filelock 3.18.0
fire 0.7.0
frozenlist 1.6.0
fsspec 2025.3.2
genson 1.3.0
gguf 0.16.3
googleapis-common-protos 1.70.0
grpcio 1.71.0
h11 0.16.0
hf-xet 1.1.0
httpcore 1.0.9
httptools 0.6.4
httpx 0.28.1
huggingface-hub 0.30.2
idna 3.10
importlib_metadata 8.0.0
interegular 0.3.3
iso3166 2.1.1
Jinja2 3.1.6
jiter 0.9.0
jsonschema 4.23.0
jsonschema-specifications 2025.4.1
lark 1.2.2
llguidance 0.7.19
llvmlite 0.44.0
lm-format-enforcer 0.10.11
lmdeploy 0.8.0
markdown-it-py 3.0.0
MarkupSafe 3.0.2
mdurl 0.1.2
mistral_common 1.5.4
mmengine-lite 0.10.7
modelscope 1.25.0
mpmath 1.3.0
msgpack 1.1.0
msgspec 0.19.0
multidict 6.4.3
nest-asyncio 1.6.0
networkx 3.4.2
ninja 1.11.1.4
numba 0.61.2
numpy 1.26.4
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-cusparselt-cu12 0.6.2
nvidia-ml-py 12.570.86
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
nvitop 1.4.2
openai 1.77.0
opencv-python-headless 4.11.0.86
opentelemetry-api 1.26.0
opentelemetry-exporter-otlp 1.26.0
opentelemetry-exporter-otlp-proto-common 1.26.0
opentelemetry-exporter-otlp-proto-grpc 1.26.0
opentelemetry-exporter-otlp-proto-http 1.26.0
opentelemetry-proto 1.26.0
opentelemetry-sdk 1.26.0
opentelemetry-semantic-conventions 0.47b0
opentelemetry-semantic-conventions-ai 0.4.6
outlines 0.1.11
outlines_core 0.1.26
packaging 25.0
partial-json-parser 0.2.1.1.post5
peft 0.14.0
pillow 11.2.1
pip 25.0
platformdirs 4.3.7
prometheus_client 0.21.1
prometheus-fastapi-instrumentator 7.1.0
propcache 0.3.1
protobuf 4.25.7
psutil 7.0.0
py-cpuinfo 9.0.0
pycountry 24.6.1
pydantic 2.11.4
pydantic_core 2.33.2
Pygments 2.19.1
pynvml 12.0.0
python-dotenv 1.1.0
python-json-logger 3.3.0
python-multipart 0.0.20
PyYAML 6.0.2
pyzmq 26.4.0
qwen-vl-utils 0.0.11
ray 2.45.0
referencing 0.36.2
regex 2024.11.6
requests 2.32.3
rich 14.0.0
rich-toolkit 0.14.5
rpds-py 0.24.0
safetensors 0.5.3
scipy 1.15.2
sentencepiece 0.2.0
setuptools 75.8.0
shellingham 1.5.4
shortuuid 1.0.13
six 1.17.0
sniffio 1.3.1
starlette 0.46.2
sympy 1.13.1
termcolor 3.1.0
tiktoken 0.9.0
timm 1.0.15
tokenizers 0.21.1
torch 2.6.0
torchaudio 2.6.0
torchvision 0.21.0
tqdm 4.67.1
transformers 4.51.3
triton 3.2.0
typer 0.15.3
typing_extensions 4.13.2
typing-inspection 0.4.0
urllib3 2.4.0
uvicorn 0.34.2
uvloop 0.21.0
vllm 0.8.5.post1
watchfiles 1.0.5
websockets 15.0.1
wheel 0.45.1
wrapt 1.17.2
xformers 0.0.29.post2
xgrammar 0.1.18
yapf 0.43.0
yarl 1.20.0
zipp 3.21.0
Error traceback
Traceback (most recent call last):
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/routing.py", line 714, in __call__
await self.middleware_stack(scope, receive, send)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/responses.py", line 262, in __call__
with collapse_excgroups():
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/contextlib.py", line 155, in __exit__
self.gen.throw(value)
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/responses.py", line 266, in wrap
await func()
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/starlette/responses.py", line 246, in stream_response
async for chunk in self.body_iterator:
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/lmdeploy/serve/openai/api_server.py", line 431, in completion_stream_generator
async for res in result_generator:
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py", line 665, in generate
prompt_input = await self._get_prompt_input(prompt,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/lmdeploy/serve/vl_async_engine.py", line 85, in _get_prompt_input
results = await self.vl_encoder.wrap_for_turbomind(results, chat_template, self.tokenizer, sequence_start)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/lmdeploy/vl/engine.py", line 103, in wrap_for_turbomind
result = self.model.to_turbomind(messages, chat_template, tokenizer, sequence_start)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/lmdeploy/vl/model/internvl.py", line 256, in to_turbomind
prompt, IMAGE_TOKEN = self.proc_messages(messages, chat_template, sequence_start)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data22/ljc/miniconda3/envs/eric/lib/python3.12/site-packages/lmdeploy/vl/model/internvl.py", line 237, in proc_messages
prompt = content[0]
~~~~~~~^^^
IndexError: list index out of range
This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue is closed because it has been stale for 5 days. Please open a new issue if you have similar issues or you have any new updates now.
I've encountered the same issue when running with InternVL. I found that the first item of the content must be a text element to bypass this problem.
In my case, if I send a system prompt and a user prompt that is a list of image URLs, it raises an error. Ensuring the first element is text resolves this issue.
