neon icon indicating copy to clipboard operation
neon copied to clipboard

pageserver: don't put layers in the heatmap if they're probably going to be compacted-away soon

Open jcsp opened this issue 1 year ago • 0 comments

Compaction runs every 20 seconds, heatmap uploads intentionally less frequent than that. The idea of heatmaps is to enable secondary tenants to prime their cache with long-existing large data, not to try and race with delta layers pending compaction.

It is not an error to include short-lived layers, but it will waste resources on the secondary locations for tenants doing intensive writes.

Some kind of check during update of the heatmap should be sufficient, such as ignoring layers that are younger than a few compaction intervals.

jcsp avatar Dec 21 '23 13:12 jcsp