metrics: Object Store

[peterbarnett03]

Object Store Metrics

Right now we serve no information on the object store performance, so this ticket looks to add some basic metrics to track.

Update the /metrics endpoint to serve the following metrics:

New Object Store Retrieval Metrics

• [ ] object_store_retrieval_duration_seconds_bucket: Time to complete object store retrievals, bucketed into 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 10, inf seconds, with types
• [ ] object_store_retrieval_duration_seconds_sum: Total time spent retrieving objects from object store
• [ ] object_store_retrieval_duration_seconds_count: Total number of object store retrieval requests

These metrics also add a bucket param, which while always the same, will make visualization easier when grouping across multiple sets.

E.g.

...
object_store_retrieval_duration_seconds_bucket{bucket="influxdb_data",le="0.05"} 2
object_store_retrieval_duration_seconds_bucket{bucket="influxdb_data",le="0.1"} 3
object_store_retrieval_duration_seconds_bucket{bucket="influxdb_data",le="0.25"} 5
...
object_store_retrieval_duration_seconds_sum{bucket="influxdb_data"} 0.68
object_store_retrieval_duration_seconds_count{bucket="influxdb_data"} 5

New Object Store Persistence Metrics

• [ ] object_store_write_duration_seconds_bucket: Time to write to object store, bucketed into 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 10, inf seconds, with types
• [ ] object_store_write_duration_seconds_sum: Total time spent writing objects to object store
• [ ] object_store_write_duration_seconds_count: Total number of object store write requests

These metrics also add a bucket param, which while always the same, will make visualization easier when grouping across multiple sets. Additionally, they add a type, which can be either parquet, wal, or none.

E.g.

...
object_store_write_duration_seconds_bucket{bucket="influxdb_data",type="parquet",le="0.05"} 2
object_store_write_duration_seconds_bucket{bucket="influxdb_data",type="parquet",le="0.1"} 3
object_store_write_duration_seconds_bucket{bucket="influxdb_data",type="parquet",le="0.25"} 5
...
object_store_write_duration_seconds_sum{bucket="influxdb_data",type="parquet"} 0.68
object_store_write_duration_seconds_count{bucket="influxdb_data",type="parquet"} 5