OpenAI endpoints don't support canceling request

[Tostino]

Using the latest development branch, I noticed in testing the OpenAI endpoints (chat_completion) that there is an issue with canceling the request if the user cancels the query for example.

If a user kills their query, the inference request is never interrupted and the inference continues unabated and further requests to the API seem to just sit there forever.

Edit: After more research, the issue is related to: https://www.postgresql.org/message-id/flat/CAFYwGJ3%2BXg7EcL2nU-MxX6p%2BO6c895Pm3mYZ-b%2B9n9DffEh5MQ%40mail.gmail.com

I saw there was another thread a couple years later, but it seems like nothing ever was done: https://www.postgresql.org/message-id/flat/CAB0yrek0C9ioVoaTiPUXy4GKzHxuf8aSc3%2BkrCCSK_Emxwf4nQ%40mail.gmail.com

So, at this point I am researching ways to use the async client to get the desired behavior:

1. It will cancel the request on the inference server when the user requests to cancel their query,
2. It will react quickly to the users request to cancel their query.

Will update when I have more info.