Add support for non streaming TGI endpoints

[nsarrazin]

Currently we only support streaming endpoints for answers, but in some cases it might make sense to use a non streaming one.

Support could be added in chat-ui/src/routes/conversation/[id]/+server.ts. You could take a look at src/lib/server/generateFromDefaultEndpoint.ts to use as an alternative for non streaming endpoints maybe ?

[kramosss]

Hi, so I believe I have a good understanding of how the current streaming endpoint works. I just wanted to verify that my understanding is correct and make sure that I modify the correct files.

• In src/routes/conversation/+server.ts the POST RequestHandler prepares the conversation and stores it in the database.
• Somewhere later in the code, the generateFromDefaultEndpoint function is called which generates answers through a streaming endpoint based on the user’s response.

My current understanding of a possible solution:

• Currently through the streaming endpoint, when the user sends a message and the AI model starts creating a response, as each part of the response is built it is sent back to the chat UI, where the user sees the response being built in real-time.
• For a non-streaming endpoint we want the AI model to process the entire message before sending back the complete response. The user will only see the response once its fully finished.
• I believe this might be able to be done by adding a boolean flag in src/routes/conversation/+server.ts to differentiate between streaming and non-streaming.
• Another typescript function will also have to be made similar to generateFromDefaultEndpoint that handles a non-streaming endpoint. Most likely it will have to take in the messages the same way from the user but will not use a for await and will return the response all at once after it has been generated.