The backend API (http://localhost:8000) stop responding with the error "upstream request timeout"

[clive97]

Hi,

I'm running Cognita with the latest commit on my local machine using Docker Compose. Everything appears to be functioning normally, with no obvious errors in the logs. I was able to create a Data Source and successfully upload PDF files to the backend. However, when I attempt to create a new collection, the API at http://localhost:8000 stops responding and returns an "upstream request timeout" error. The API only resumes functioning once the backend has finished parsing the files and completes the collection creation process.

While this isn't a major issue in a testing environment, where I can do other tasks while waiting, it becomes problematic in a production environment. Multiple users might be accessing the portal simultaneously to perform different tasks, and when the API freezes, much of the portal becomes unusable, as most of its features rely on the APIs.

Additionally, when attempting to parse a larger batch of files, the process takes so long that it fails the health check, causing the orchestrator (e.g. AKS, ECS, Kubernetes) to restart the container.

Is this behavior expected, or is there something I might have missed?

Thanks, Clive