prometheus_client_php icon indicating copy to clipboard operation
prometheus_client_php copied to clipboard

APCng storage - avoid worst-case memory usage in buildPermutationTree

Open Princemachiavelli opened this issue 2 years ago • 2 comments

The peak memory usage of the APCng storage adapter while collecting the metrics for display is currently always the worst-case for the cardinality of the metric label values.

This change replaces the buildPermutationTree implementation with a (cartesian product) generator which avoids keeping every metric label value permutation in memory at once. This can reduce peak memory usage by orders of magnitude for metrics with high cardinality but sparse actual usage. This helps avoid exceeding the PHP memory_limit.

Besides the reduced memory usage, I haven't seen much of a difference in how long buildPermutationTree takes to run. (To improve that, only the actually used metric label pairs could be to tracked separately to avoid building the permutations completely.)

Princemachiavelli avatar Jun 21 '23 23:06 Princemachiavelli

This would be great to land, we are hitting the PHP limit as well.

mlebrun avatar Jul 27 '23 21:07 mlebrun

I've tested a copy of this PR and it works well to reduce memory usage with histograms

xocasdashdash avatar Sep 03 '24 16:09 xocasdashdash