Handling large message/result set sizes in search

[dennisoelkers]

When retrieving and displaying a large result set of messages (as in 150*n mb) during search, we need to make sure that:

• We do not exceed the configured http client buffer size (resulting in entity content is too long [...] for the configured buffer limit [...] exceptions otherwise, where the default is 100Mb)
• We do not overload the browser during rendering

Potential strategies could include:

• Always retrieve truncated summaries for messages when fetching result sets for multiple messages
• Retrieving complete messages only when showing message details and only on demand (when expanding a message)

[boosty]

If we would ever want to enable the indexing and processing failure handling in Graylog Enterprise / Cloud by default, I think this issue would be a prerequisite.

[BBruce-Graylog]

Following up on the status of this issue. @boosty @bernd

[boosty]

Hi @BBruce-Graylog, this has been assigned to the Search Team, so I would ask @dennisoelkers to comment on this.

[BBruce-Graylog]

@dennisoelkers Wanted to see if there was any update on this issue.

[kmerz]

@stefanthegerman, can you tell us how important this issue still is? We need help prioritizing it.