elasticsearch icon indicating copy to clipboard operation
elasticsearch copied to clipboard

Published 20 hours ago • verified publisher icon elastic

Protect get-snapshots APIs against excessive memory use

Open DaveCTurner opened this issue 1 year ago • 2 comments

The get-snapshots APIs accumulate the details of all matching snapshots in memory before responding. In a high-index-count cluster with a long retention period on snapshots this can add up to GiB+ of heap on the elected master to request all snapshots.

We should make this API more efficient for sure, see https://github.com/elastic/elasticsearch/issues/95345, but we should also protect the cluster against accidental use of this API to request too many snapshots by rejecting such requests before they can cause harm.

DaveCTurner avatar Jan 22 '24 08:01 DaveCTurner

Pinging @elastic/es-distributed (Team:Distributed)

elasticsearchmachine avatar Jan 22 '24 08:01 elasticsearchmachine

In fact once https://github.com/elastic/elasticsearch/pull/104851 is done we could also make this API run in near-constant memory, streaming its results in chunks all the way from the master node to the client.

DaveCTurner avatar Feb 16 '24 08:02 DaveCTurner