Synchronisation issue in Buffer causing metrics scraping failure

[markusaaltosc]

trafficstars

Fixed a synchronisation issue in Buffer where changes made by the doAppend() method may not always be visible to the run() method. With the bug the implementation was stuck in busy loop stopping the metrics scraping. The issue surfaced when running in AWS Graviton ARM processor with 5s scraping interval and large number of simultaneous threads constantly adding new observations. Once fixed and the doAppend changes becoming visible it was also possible the bufferPos to be larger than expectedBufferSize (if doAppend() was called after expectedBufferSize was calculated and before appendLock was acquired in runMethod) and that is now also fixed by ensuring we stay in the loop only if bufferPos < expectedBufferSize. Please @fstab check this.

[fstab]

Thanks a lot @markusaaltosc, I'll have a look.

We did extensive long-running load tests on Intel / AMD64 CPUs, but never on Graviton ARM.

I'll get back to you as soon as I found the time to review.

[markusaaltosc]

We have now been running with the fix for few weeks already both on Intel and Graviton so looking stable so far.

[zeitlinger]

@markusaaltosc I've created an alternative fix - https://github.com/prometheus/client_java/pull/991 - please have a look