druid
druid copied to clipboard
Emit disk spill and merge buffer utilisation metrics for GroupBy queries
This change is to emit following metrics as part of GroupByStatsMonitor
monitor,
mergeBuffer/usedCount
-> Number of merge buffers used.
mergeBuffer/acquisitionTimeNs
-> Total time required to acquire merge buffer.
mergeBuffer/acquisitionCount
-> Number of times merge buffer were acquired.
groupBy/spilledQueries
-> Number of queries that spilled onto the disk.
groupBy/spilledBytes
-> Spilled bytes on the disk.
Testing
- Added UT to verify that
GroupByStatsMonitor
is reporting the new metrics. - Added UT for
GroupByStatsProvider
. - Tested metric emission in a local druid cluster for a group by query.
2024-10-16T17:51:11,277 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"query/failed/count","service":"druid/historical","host":"localhost:8083","version":"32.0.0-SNAPSHOT","value":0,"timestamp":"2024-10-16T17:51:11.277Z"}
2024-10-16T17:51:11,277 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"mergeBuffer/pendingRequests","service":"druid/historical","host":"localhost:8083","version":"32.0.0-SNAPSHOT","value":0,"timestamp":"2024-10-16T17:51:11.277Z"}
2024-10-16T17:51:11,277 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"mergeBuffer/acquiredCount","service":"druid/historical","host":"localhost:8083","version":"32.0.0-SNAPSHOT","value":0,"timestamp":"2024-10-16T17:51:11.277Z"}
2024-10-16T17:51:11,277 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"groupBy/acquisitionTimeNs","service":"druid/historical","host":"localhost:8083","version":"32.0.0-SNAPSHOT","value":1351083,"timestamp":"2024-10-16T17:51:11.277Z"}
2024-10-16T17:51:11,277 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"groupBy/spilledBytes","service":"druid/historical","host":"localhost:8083","version":"32.0.0-SNAPSHOT","value":4013,"timestamp":"2024-10-16T17:51:11.277Z"}
2024-10-16T17:51:11,766 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"groupBy/acquisitionTimeNs","service":"druid/broker","host":"localhost:8082","version":"32.0.0-SNAPSHOT","value":5304583,"timestamp":"2024-10-16T17:51:11.765Z"}
2024-10-16T17:51:11,766 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"groupBy/spilledBytes","service":"druid/broker","host":"localhost:8082","version":"32.0.0-SNAPSHOT","value":12373,"timestamp":"2024-10-16T17:51:11.766Z"}
2024-10-16T17:51:11,766 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"service/heartbeat","service":"druid/broker","host":"localhost:8082","version":"32.0.0-SNAPSHOT","value":1,"timestamp":"2024-10-16T17:51:11.766Z"}
2024-10-16T17:51:11,766 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"jetty/numOpenConnections","service":"druid/broker","host":"localhost:8082","version":"32.0.0-SNAPSHOT","value":1,"timestamp":"2024-10-16T17:51:11.766Z"}
2024-10-16T17:51:11,766 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - [metrics] {"feed":"metrics","metric":"jetty/threadPool/total","service":"druid/broker","host":"localhost:8082","version":"32.0.0-SNAPSHOT","value":58,"timestamp":"2024-10-16T17:51:11.766Z"}