prometheus_client_php icon indicating copy to clipboard operation
prometheus_client_php copied to clipboard

Bad performance after updating to v2.6.1

Open timmipetit opened this issue 3 years ago • 7 comments

After updating to v2.6.1, which introduced https://github.com/PromPHP/prometheus_client_php/pull/91, we've seen a enormous drop in performance for our collection script. This is interesting, because the merge request is meant to increase performance.

Here's a screenshot from the average response time of our prometheus collection script: Screenshot 2022-06-17 at 11 50 11

timmipetit avatar Jun 17 '22 10:06 timmipetit

Hey @timmipetit,

can you try to downgrade to v2.6.0 just to make sure it is really related to the script? How many metrics do you expose? Might it be that the redis has a lot of keys?

LKaemmerling avatar Jun 17 '22 20:06 LKaemmerling

Hi @LKaemmerling. We've downgraded back to v2.6.0, and performance is back to normal for us. We don't expose a lot of metrics, 6 histograms (with 16 buckets each) and 3 gauges (so no summaries). But our Redis instance is also used for sessions though, so we do have a lot of other keys (400k or so). The keys used for sessions have a different prefix. But since scan will iterate all keys, it will fetch all session keys (before discarding them again because they don't match)

timmipetit avatar Jun 20 '22 07:06 timmipetit

I have to say the same from project https://supportbox.com/ - after upgrade to 2.6.1 so bad performance. Wipe storage (Redis) does not help.

grogy avatar Jun 29 '22 11:06 grogy

Hello together,

i just released v2.6.2 with the revert of the given MR.

LKaemmerling avatar Jun 30 '22 03:06 LKaemmerling

Hey @grogy & @timmipetit,

would it be possible that you test https://github.com/PromPHP/prometheus_client_php/pull/99/files against your workload (new StorageEngine RedisNG). This basically fixes the usage of KEYS and SCAN.

LKaemmerling avatar Nov 29 '22 05:11 LKaemmerling

@LKaemmerling Thanks for the heads up. We'll test it

timmipetit avatar Nov 30 '22 11:11 timmipetit

Tested on our workload, works fine!

timmipetit avatar Dec 01 '22 11:12 timmipetit