FastChat icon indicating copy to clipboard operation
FastChat copied to clipboard

Published 20 hours ago • verified publisher icon lm-sys

OpenAI style API : Object of type SeparatorStyle is not JSON serializable

Open Extremys opened this issue 2 years ago • 1 comments

Hello, I've deployed the OpenAI style API in a container when trying to use through OpenAI package I get this kind of error, any idea?

The command line: python -m fastchat.serve.model_worker --model-name "gpt-3.5-turbo,text-davinci-003,text-embedding-ada-002,vicuna-13b-v1.1" --model-path ./vicuna-13B-v1.1/ --load-8bit

The code use:

import openai
import os

os.environ['REQUESTS_CA_BUNDLE'] = './mycert.crt'
openai.api_key = "EMPTY" # Not support yet
openai.api_base = "https://myurl/v1"

model = "vicuna-13b-v1.1"
prompt = "Once upon a time"

# create a completion
completion = openai.Completion.create(model=model, prompt=prompt, max_tokens=64)
# print the completion
print(prompt + completion.choices[0].text)

# create a chat completion
completion = openai.ChatCompletion.create(
  model=model,
  messages=[{"role": "user", "content": "Hello! What is your name?"}]
)
# print the completion
print(completion.choices[0].message.content)

embedding = openai.Embedding.create(
    input="Your text goes here", model="text-embedding-ada-002"
)["data"][0]["embedding"]
len(embedding)
print(embedding)

The error I get:

2023-06-12 18:14:22 | INFO | stdout | INFO:     ::1:41410 - "POST /worker_get_conv_template HTTP/1.1" 500 Internal Server Error
2023-06-12 18:14:22 | ERROR | stderr | ERROR:    Exception in ASGI application
2023-06-12 18:14:22 | ERROR | stderr | Traceback (most recent call last):
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
2023-06-12 18:14:22 | ERROR | stderr |     result = await app(self.scope, self.receive, self.send)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
2023-06-12 18:14:22 | ERROR | stderr |     return await self.app(scope, receive, send)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/fastapi/applications.py", line 208, in __call__
2023-06-12 18:14:22 | ERROR | stderr |     await super().__call__(scope, receive, send)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
2023-06-12 18:14:22 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
2023-06-12 18:14:22 | ERROR | stderr |     raise exc from None
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
2023-06-12 18:14:22 | ERROR | stderr |     await self.app(scope, receive, _send)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
2023-06-12 18:14:22 | ERROR | stderr |     raise exc from None
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
2023-06-12 18:14:22 | ERROR | stderr |     await self.app(scope, receive, sender)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 580, in __call__
2023-06-12 18:14:22 | ERROR | stderr |     await route.handle(scope, receive, send)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 241, in handle
2023-06-12 18:14:22 | ERROR | stderr |     await self.app(scope, receive, send)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 52, in app
2023-06-12 18:14:22 | ERROR | stderr |     response = await func(request)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/fastapi/routing.py", line 250, in app
2023-06-12 18:14:22 | ERROR | stderr |     response = actual_response_class(response_data, **response_args)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/responses.py", line 53, in __init__
2023-06-12 18:14:22 | ERROR | stderr |     self.body = self.render(content)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/responses.py", line 161, in render
2023-06-12 18:14:22 | ERROR | stderr |     return json.dumps(
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/json/__init__.py", line 234, in dumps
2023-06-12 18:14:22 | ERROR | stderr |     return cls(
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/json/encoder.py", line 199, in encode
2023-06-12 18:14:22 | ERROR | stderr |     chunks = self.iterencode(o, _one_shot=True)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/json/encoder.py", line 257, in iterencode
2023-06-12 18:14:22 | ERROR | stderr |     return _iterencode(o, 0)
2023-06-12 18:14:22 | ERROR | stderr |   File "/opt/conda/lib/python3.9/json/encoder.py", line 179, in default
2023-06-12 18:14:22 | ERROR | stderr |     raise TypeError(f'Object of type {o.__class__.__name__} '
2023-06-12 18:14:22 | ERROR | stderr | TypeError: Object of type SeparatorStyle is not JSON serializable

Extremys avatar Jun 12 '23 18:06 Extremys

it seems related to the fact that I use "https" with certificate because in "http" it's working perfectly, any plan to support it?

2023-06-19 08:51:35 | INFO | model_worker | Send heart beat. Models: ['gpt-3.5-turbo', 'text-davinci-003', 'text-embedding-ada-002', 'vicuna-13b-v1.1']. Semaphore: None. global_counter: 0. worker_id: 8c3723.
2023-06-19 08:51:35 | INFO | controller | Receive heart beat. http://localhost:21002
2023-06-19 08:51:35 | INFO | stdout | INFO:     ::1:37590 - "POST /receive_heart_beat HTTP/1.1" 200 OK
2023-06-19 08:51:46 | INFO | controller | names: ['http://localhost:21002'], queue_lens: [0.0], ret: http://localhost:21002
2023-06-19 08:51:46 | INFO | stdout | INFO:     ::1:53542 - "POST /get_worker_address HTTP/1.1" 200 OK
2023-06-19 08:51:46 | INFO | controller | names: ['http://localhost:21002'], queue_lens: [1.0], ret: http://localhost:21002
2023-06-19 08:51:46 | INFO | stdout | INFO:     ::1:53552 - "POST /get_worker_address HTTP/1.1" 200 OK
2023-06-19 08:51:46 | INFO | stdout | INFO:     ::1:56264 - "POST /worker_get_conv_template HTTP/1.1" 500 Internal Server Error
2023-06-19 08:51:46 | ERROR | stderr | ERROR:    Exception in ASGI application
2023-06-19 08:51:46 | ERROR | stderr | Traceback (most recent call last):
2023-06-19 08:51:46 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 373, in run_asgi
2023-06-19 08:51:46 | ERROR | stderr |     result = await app(self.scope, self.receive, self.send)
2023-06-19 08:51:46 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
2023-06-19 08:51:46 | ERROR | stderr |     return await self.app(scope, receive, send)
2023-06-19 08:51:46 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/fastapi/applications.py", line 208, in __call__
2023-06-19 08:51:46 | ERROR | stderr |     await super().__call__(scope, receive, send)
2023-06-19 08:51:46 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
2023-06-19 08:51:46 | ERROR | stderr |     await self.middleware_stack(scope, receive, send)
2023-06-19 08:51:46 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
INFO:     10.9.197.179:34046 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2023-06-19 08:51:46 | ERROR | stderr |     raise exc from None
2023-06-19 08:51:46 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
2023-06-19 08:51:46 | ERROR | stderr |     await self.app(scope, receive, _send)
2023-06-19 08:51:46 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
2023-06-19 08:51:46 | ERROR | stderr |     raise exc from None
2023-06-19 08:51:46 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
2023-06-19 08:51:46 | ERROR | stderr |     await self.app(scope, receive, sender)
2023-06-19 08:51:46 | ERROR | stderr |   File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 580, in __call__
2023-06-19 08:51:46 | ERROR | stderr |     await route.handle(scope, receive, send)

Extremys avatar Jun 13 '23 18:06 Extremys

solved by making the key field effectively empty

openai.api_key = '' # Not support yet

Extremys avatar Jul 06 '23 05:07 Extremys