lakeFS icon indicating copy to clipboard operation
lakeFS copied to clipboard

Monitor API (lakeFS and S3 gateway) and blockstore concurrency

Open arielshaqed opened this issue 5 months ago • 0 comments

Add started/finished counters (or maybe gauges) to monitor current concurrency on both app-facing and storage-facing sides. So probably 2 pairs of started/finished counters, one for incoming requests (lakeFS and S3 gateway APIs) and one for outbound requests to storage (block adapters, at least S3). Label them hopefully nicely. This will help diagnose some issues. (Concurrency is the difference finished-started)

Relevant to #9128 (but completely different).

Example of an issue for which this could help

We are able to see (under synthetic load -- probably not possible for users) episodes with some strangely high processing latencies, quite possibly due to concurrency on the S3 adapter. Knowing concurrency should help us understand these episodes.

arielshaqed avatar Jun 09 '25 06:06 arielshaqed