druid icon indicating copy to clipboard operation
druid copied to clipboard

Emit disk spill and merge buffer utilisation metrics for GroupBy queries

Open findingrish opened this issue 4 months ago • 0 comments

This change is to emit following metrics as part of GroupByStatsMonitor monitor, mergeBuffer/usedCount -> Number of merge buffers used. mergeBuffer/acquisitionTimeNs -> Total time required to acquire merge buffer. mergeBuffer/acquisitionCount -> Number of times merge buffer were acquired. groupBy/spilledQueries -> Number of queries that spilled onto the disk. groupBy/spilledBytes-> Spilled bytes on the disk.

Testing

  1. Added UT to verify that GroupByStatsMonitor is reporting the new metrics.
  2. Added UT for GroupByStatsProvider.
  3. Tested metric emission in a local druid cluster for a group by query.
2024-10-16T17:51:11,277 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"query/failed/count","service":"druid/historical","host":"localhost:8083","version":"32.0.0-SNAPSHOT","value":0,"timestamp":"2024-10-16T17:51:11.277Z"}
2024-10-16T17:51:11,277 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"mergeBuffer/pendingRequests","service":"druid/historical","host":"localhost:8083","version":"32.0.0-SNAPSHOT","value":0,"timestamp":"2024-10-16T17:51:11.277Z"}
2024-10-16T17:51:11,277 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"mergeBuffer/acquiredCount","service":"druid/historical","host":"localhost:8083","version":"32.0.0-SNAPSHOT","value":0,"timestamp":"2024-10-16T17:51:11.277Z"}
2024-10-16T17:51:11,277 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"groupBy/acquisitionTimeNs","service":"druid/historical","host":"localhost:8083","version":"32.0.0-SNAPSHOT","value":1351083,"timestamp":"2024-10-16T17:51:11.277Z"}
2024-10-16T17:51:11,277 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"groupBy/spilledBytes","service":"druid/historical","host":"localhost:8083","version":"32.0.0-SNAPSHOT","value":4013,"timestamp":"2024-10-16T17:51:11.277Z"}
2024-10-16T17:51:11,766 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"groupBy/acquisitionTimeNs","service":"druid/broker","host":"localhost:8082","version":"32.0.0-SNAPSHOT","value":5304583,"timestamp":"2024-10-16T17:51:11.765Z"}
2024-10-16T17:51:11,766 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"groupBy/spilledBytes","service":"druid/broker","host":"localhost:8082","version":"32.0.0-SNAPSHOT","value":12373,"timestamp":"2024-10-16T17:51:11.766Z"}
2024-10-16T17:51:11,766 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"service/heartbeat","service":"druid/broker","host":"localhost:8082","version":"32.0.0-SNAPSHOT","value":1,"timestamp":"2024-10-16T17:51:11.766Z"}
2024-10-16T17:51:11,766 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"jetty/numOpenConnections","service":"druid/broker","host":"localhost:8082","version":"32.0.0-SNAPSHOT","value":1,"timestamp":"2024-10-16T17:51:11.766Z"}
2024-10-16T17:51:11,766 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"jetty/threadPool/total","service":"druid/broker","host":"localhost:8082","version":"32.0.0-SNAPSHOT","value":58,"timestamp":"2024-10-16T17:51:11.766Z"}

findingrish avatar Oct 16 '24 05:10 findingrish