lakeFS
lakeFS copied to clipboard
Monitor API (lakeFS and S3 gateway) and blockstore concurrency
Add started/finished counters (or maybe gauges) to monitor current concurrency on both app-facing and storage-facing sides. So
probably 2 pairs of started/finished counters, one for incoming requests (lakeFS and S3 gateway APIs) and one for outbound
requests to storage (block adapters, at least S3). Label them hopefully nicely. This will help
diagnose some issues. (Concurrency is the difference finished-started)
Relevant to #9128 (but completely different).
Example of an issue for which this could help
We are able to see (under synthetic load -- probably not possible for users) episodes with some strangely high processing latencies, quite possibly due to concurrency on the S3 adapter. Knowing concurrency should help us understand these episodes.