[BUG] Slow retrieval of documents

[jeremylatorre]

Describe the bug

When using RAG, I noticed that performances where very low when trying to retrieve information from documents. Any lead about what is causing this. Clearly the difference between rag and standard bot is huge at that time. It might also depend on number of documents.

[statefb]

Could you provide the detail? Please do not ignore the issue template.

[jeremylatorre]

When I use my bot, it takes sometimes 17s before starting to render the content.

The first step [Retrieve Knowledge] looks ok, but after that, cursor stay at the beginning of the line for a very long time before starting to write the response.

This doesn't happen when using the LLM without documents, and it seems to increase with the number of document in the knowledge base.

[statefb]

Could you check the items in postgres? We can use management console to query and the table definition can be reffered here.

[jeremylatorre]

Could it be related to the instantion of a Lambda function? The behavior only appear on the first inference for a new conversation

[statefb]

Have you forked and customized this sample? Large container causes longer cold start. You can check whether the invocation is cold or not by reffering cloudwatch logs.
https://stackoverflow.com/questions/47061146/how-can-i-detect-cold-starts-in-aws-lambda

[jeremylatorre]

Ok I have digged a llittle bit on the logs and I found that the POST call to related-documents is very long by itself:

Request POST /conversation/related-documents (15:15:32 - 15:15:44) Duration : 11697 ms

I'm not currently up to date with the codebase and we will perform a full update soon. I'll keep you updated on this.

[statefb]

@jeremylatorre Thank you for the update. The query part to postgres seems to making the very long latency. For the cost effectiveness, the ACU configured at minimum value by default. Choosing larger value might solve the problem .

[github-actions[bot]]

This issue has been labeled as "stale" due to no response by the reporter within 1 month (and 14 days after last commented by someone). And it will be closed automatically 14 days later if not responded.

[github-actions[bot]]

This issue has been closed due to no response within 14 days after labeled as "stale", 14 days after last reopened, and 14 days after last commented.