message bucket: collect messages and send in a single prompt

[avelino]

The POST /chat/{chat_id} receives one message at a time, when it receives it, it processes it in the llm and returns a message to you, example:

• QA 1: hello how are you?
• QA 2: I need help, can you help me, please?
• QA 3: how do I perform operation XYZ on the web page?

This flow of conversation is common in a chat environment (e.g. WhatsApp), where the user breaks the line. What the user would like to receive is only the answer to QA 3, the other messages are introductory ("presentation").

As implemented today, we answer one message at a time:

• Reply from QA1: Hi, how can I help?
• Reply from QA2: It would be a pleasure to help you, how can I help you?
• Reply from QA3: You should access, ... the answer to the question

the answer that matters is QA3, QA1 and QA2 are ~~"duplicated"~~

solution

Parameter in the endpoint (POST, create message) called message bucket, which activates intelligence to collect messages in the backend and make a single call to the LLM sending the collection of messages.

I can think of a solution to collect requests and if no message is received at X after the last message received, call the LLM aggregating all the messages not sent.

it's not the best solution, but it's the solution that comes to mind at first - this issue is to discuss the best solution, probably the proposed solution is not the best