pydantic.error_wrappers.ValidationError: 3 validation errors for ChatCompletionResponse

[talhaanwarch]

System Info

Platform : Ubuntu Python 3.10

Information

• [X] The official example notebooks/scripts
• [ ] My own modified scripts

Related Components

• [ ] backend
• [ ] bindings
• [ ] python-bindings
• [ ] chat-ui
• [ ] models
• [ ] circleci
• [ ] docker
• [X] api

Reproduction

parameters body to /v1/chat/completions endpoint

{
  "model": "ggml-llama-2-7b-chat-q4_0.bin",
  "messages": [
    {
      "role": "user",
      "content": "what is AI?"
    }
  ]
}

Throw following error

gpt4all_api    | INFO:     172.20.0.1:47342 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
gpt4all_api    | ERROR:    Exception in ASGI application
gpt4all_api    | Traceback (most recent call last):
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 436, in run_asgi
gpt4all_api    |     result = await app(  # type: ignore[func-returns-value]
gpt4all_api    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
gpt4all_api    |     return await self.app(scope, receive, send)
gpt4all_api    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 289, in __call__
gpt4all_api    |     await super().__call__(scope, receive, send)
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
gpt4all_api    |     await self.middleware_stack(scope, receive, send)
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
gpt4all_api    |     raise exc
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
gpt4all_api    |     await self.app(scope, receive, _send)
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 91, in __call__
gpt4all_api    |     await self.simple_response(scope, receive, send, request_headers=headers)
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 146, in simple_response
gpt4all_api    |     await self.app(scope, receive, send)
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
gpt4all_api    |     raise exc
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
gpt4all_api    |     await self.app(scope, receive, sender)
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
gpt4all_api    |     raise e
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
gpt4all_api    |     await self.app(scope, receive, send)
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
gpt4all_api    |     await route.handle(scope, receive, send)
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
gpt4all_api    |     await self.app(scope, receive, send)
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
gpt4all_api    |     response = await func(request)
gpt4all_api    |                ^^^^^^^^^^^^^^^^^^^
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 273, in app
gpt4all_api    |     raw_response = await run_endpoint_function(
gpt4all_api    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api    |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 190, in run_endpoint_function
gpt4all_api    |     return await dependant.call(**values)
gpt4all_api    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api    |   File "/app/api_v1/routes/chat.py", line 55, in chat_completion
gpt4all_api    |     return ChatCompletionResponse(
gpt4all_api    |            ^^^^^^^^^^^^^^^^^^^^^^^
gpt4all_api    |   File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
gpt4all_api    | pydantic.error_wrappers.ValidationError: 3 validation errors for ChatCompletionResponse
gpt4all_api    | choices -> 0 -> message
gpt4all_api    |   field required (type=value_error.missing)
gpt4all_api    | choices -> 0 -> index
gpt4all_api    |   field required (type=value_error.missing)
gpt4all_api    | choices -> 0 -> finish_reason
gpt4all_api    |   field required (type=value_error.missing)

where as following parameter to /v1/completions/ work with very poor results.

{
  "model": "ggml-llama-2-7b-chat-q4_0.bin",
  "prompt": [
    "What is AI"
  ],
  "max_tokens": 100,
  "temperature": 0.18,
  "top_p": 1,
  "top_k": 50,
  "n": 1,
  "stream": false,
  "repeat_penalty": 1.18
}

Expected behavior

Expecting it not give error

[razorrider7]

I am also receiving this error with the Docker Container which includes FastAPI.

[Oleh-Hudzo]

have the same problem now. have there been any fixes?

[aaad]

I have the same problem.

[SkyWriter]

This is the relevant code:

@router.post("/completions", response_model=ChatCompletionResponse)
async def chat_completion(request: ChatCompletionRequest):
    '''
    Completes a GPT4All model response.
    '''

    return ChatCompletionResponse(
        id='asdf',
        created=time.time(),
        model=request.model,
        choices=[{}],
        usage={'prompt_tokens': 0, 'completion_tokens': 0, 'total_tokens': 0},
    )

choices here are expected to be like:

{
  "index": 0,
  "message": {
    "role": "assistant",
    "content": "\n\nHello there, how may I assist you today?",
  },
  "finish_reason": "stop"
}

In addition to that, no actual model is being called while handling the request.

My guess that it's not a bug, but rather a missing implementation altogether.

[timothyckl]

Also facing this error while using Mistral models for metadata extraction :(