Quickwit consumes a lot of RAM when indexing on 1000 indexes and more

[fmassot]

An indexer can take more than 15GB; it was observed on a k8s cluster, which has 1000 indexes.

Possible explanation: we have several queues in a given indexing pipeline, and each queue can stack up to X messages. For example, the queue size of the doc processor is 10, and given that the source is generating a message of size 1MB, we can deduce it will take 10GB if doc processor queues are full for all indexes...

We should first confirm this is the correct explanation.

Possible solution: define a RAM budget at the server level and use it to limit the number of messages waiting in the doc processor queue (or other actors?). The budget could be equivalent to the ingest API memory budget.