druid icon indicating copy to clipboard operation
druid copied to clipboard

Restore Sink Metric Emission Behaviour: Emit them per-Sink instead of per-FireHydrant

Open findingrish opened this issue 5 months ago • 0 comments

This change https://github.com/apache/druid/pull/15757 to merge FireHydrants flatly for realtime queries for optimising memory usage led to metrics like query/segment/time getting emitted per-FireHydrant instead of per-Sink.

This change restores the metric emission behaviour while keeping the optimisation intact.

It introduces a new SinkMetricsEmittingQueryRunner which accumulates the FireHydrants metrics per-Sink and emits them in the end. The emitted metrics are query/segment/time, query/segmentAndCacheTime & query/wait/time. query/wait/time is the time taken to start processing the first FireHydrant for the sink.

Testing

  • Added UT to verify that each of the above metrics are emitted once per Sink.
  • Locally verified the changes for realtime ingestion. Sample emitted metric,
{"type":"scan","version":"32.0.0-SNAPSHOT","duration":"PT86400S","feed":"metrics","metric":"query/segment/time","hasFilters":"false","service":"druid/middleManager","segment":"kttm_2019-08-25T04:00:00.000Z_2019-08-25T05:00:00.000Z_2024-09-26T15:28:40.091Z","host":"localhost:8100","context":{"defaultTimeout":300000,"finalize":false,"maxQueuedBytes":5242880,"maxScatterGatherBytes":9223372036854775807,"queryFailTime":1727364858115,"queryId":"8dab0179-ec39-4f5f-a9e2-9039da416a5a","queryResourceId":"14c72b21-2dbb-471b-91c0-d7fbe11414b5","scanOutermost":false,"sqlOuterLimit":1001,"sqlQueryId":"8dab0179-ec39-4f5f-a9e2-9039da416a5a","sqlStringifyArrays":false,"timeout":299997},"interval":["2019-08-25T00:00:00.000Z/2019-08-26T00:00:00.000Z"],"id":"8dab0179-ec39-4f5f-a9e2-9039da416a5a","value":0,"dataSource":"kttm","timestamp":"2024-09-26T15:29:18.215Z"}

findingrish avatar Sep 26 '24 17:09 findingrish