bug: when use backend=pt in normal or adapter mode,RuntimeError: Exception caught during generation: Failed to parse JSON from SSE message:(vllm is ok)

[baibaiw5]

Describe the bug

when use backend=pt in normal or adapter mode and send a request to openvllm,and error will occur, but if i change backend=vllm,everything is ok

To reproduce

1.openllm start /home/bmb/models/facebook/opt-125m --device 0 -p 8901 --dtype float16 --backend pt 2.openllm query --endpoint http://0.0.0.0:8901 --timeout 120 "What is the weight of the earth?"

Logs

---------------------------------------error log
Traceback (most recent call last):
File "/home/bmb/anaconda3/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi
result = await app(  # type: ignore[func-returns-value]
File "/home/bmb/anaconda3/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
return await self.app(scope, receive, send)
File "/home/bmb/anaconda3/lib/python3.10/site-packages/uvicorn/middleware/message_logger.py", line 86, in __call__
raise exc from None
File "/home/bmb/anaconda3/lib/python3.10/site-packages/uvicorn/middleware/message_logger.py", line 82, in __call__
await self.app(scope, inner_receive, inner_send)
File "/home/bmb/anaconda3/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/bmb/anaconda3/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
raise exc
File "/home/bmb/anaconda3/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
await self.app(scope, receive, _send)
File "/home/bmb/anaconda3/lib/python3.10/site-packages/bentoml/_internal/server/http/traffic.py", line 26, in __call__
await self.app(scope, receive, send)
File "/home/bmb/anaconda3/lib/python3.10/site-packages/bentoml/_internal/server/http/instruments.py", line 135, in __call__
await self.app(scope, receive, wrapped_send)
File "/home/bmb/anaconda3/lib/python3.10/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 596, in __call__
await self.app(scope, otel_receive, otel_send)
File "/home/bmb/anaconda3/lib/python3.10/site-packages/bentoml/_internal/server/http/access.py", line 126, in __call__
await self.app(scope, receive, wrapped_send)
File "/home/bmb/anaconda3/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
raise exc
File "/home/bmb/anaconda3/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
await self.app(scope, receive, sender)
File "/home/bmb/anaconda3/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
await route.handle(scope, receive, send)
File "/home/bmb/anaconda3/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/home/bmb/anaconda3/lib/python3.10/site-packages/starlette/routing.py", line 69, in app
await response(scope, receive, send)
File "/home/bmb/anaconda3/lib/python3.10/site-packages/starlette/responses.py", line 270, in __call__
async with anyio.create_task_group() as task_group:
File "/home/bmb/anaconda3/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 597, in __aexit__
raise exceptions[0]
File "/home/bmb/anaconda3/lib/python3.10/site-packages/starlette/responses.py", line 273, in wrap
await func()
File "/home/bmb/anaconda3/lib/python3.10/site-packages/starlette/responses.py", line 262, in stream_response
async for chunk in self.body_iterator:
File "/data/bmb/anaconda3/lib/python3.10/site-packages/openllm/_service.py", line 28, in generate_stream_v1
async for it in llm.generate_iterator(**llm_model_class(**input_dict).model_dump()):
File "/home/bmb/anaconda3/lib/python3.10/site-packages/openllm/_llm.py", line 125, in generate_iterator
raise RuntimeError(f'Exception caught during generation: {err}') from err
RuntimeError: Exception caught during generation: Failed to parse JSON from SSE message: '{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\n","token_ids":[50118],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe","token_ids":[50118,133],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight","token_ids":[50118,133,2408],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of","token_ids":[50118,133,2408,9],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of the","token_ids":[50118,133,2408,9,5],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of the earth","token_ids":[50118,133,2408,9,5,6872],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of the earth is","token_ids":[50118,133,2408,9,5,6872,16],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of the earth is equal","token_ids":[50118,133,2408,9,5,6872,16,3871],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of the earth is equal to","token_ids":[50118,133,2408,9,5,6872,16,3871,7],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of the earth is equal to the","token_ids":[50118,133,2408,9,5,6872,16,3871,7,5],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of the earth is equal to the weight","token_ids":[50118,133,2408,9,5,6872,16,3871,7,5,2408],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of the earth is equal to the weight of","token_ids":[50118,133,2408,9,5,6872,16,3871,7,5,2408,9],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of the earth is equal to the weight of the","token_ids":[50118,133,2408,9,5,6872,16,3871,7,5,2408,9,5],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of the earth is equal to the weight of the sun","token_ids":[50118,133,2408,9,5,6872,16,3871,7,5,2408,9,5,3778],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of the earth is equal to the weight of the sun.","token_ids":[50118,133,2408,9,5,6872,16,3871,7,5,2408,9,5,3778,4],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":false,"outputs":[{"index":0,"text":"\\nThe weight of the earth is equal to the weight of the sun.","token_ids":[50118,133,2408,9,5,6872,16,3871,7,5,2408,9,5,3778,4,2],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":null}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}{"prompt":"","finished":true,"outputs":[{"index":0,"text":"\\nThe weight of the earth is equal to the weight of the sun.","token_ids":[2,2264,16,5,2408,9,5,6872,116,50118,133,2408,9,5,6872,16,3871,7,5,2408,9,5,3778,4,2],"cumulative_logprob":0.0,"logprobs":null,"finish_reason":"stop"}],"prompt_token_ids":[2,2264,16,5,2408,9,5,6872,116],"prompt_logprobs":null,"request_id":"openllm-fc35ab24576a4b52aa088c60b745fac3"}'
^C^C^CException ignored in: <module 'threading' from '/home/bmb/anaconda3/lib/python3.10/threading.py'>
Traceback (most recent call last):
  File "/home/bmb/anaconda3/lib/python3.10/threading.py", line 1567, in _shutdown
    lock.acquire()

Environment

openllm 0.4.41 torch 2.1.2 sentence-transformers 2.2.2 transformers 4.36.0 transformers-stream-generator 0.0.4

System information (Optional)

GPU:A6000 system:ubuntu 22.04

[seanbenhur]

+1, I am also getting the same error

[linevers1926]

+1

[bojiang]

close for openllm 0.6