Add QueryEventListener for writing Pinot Event Data to DeepStore

[anilsomisetty]

Currently, Pinot does not have a BrokerQueryEventListener implementation. This feature request suggests adding a QueryEventListener that will write Pinot event data to a configured DeepStore.

DeepStore Pinot event data can then be ingested back into Pinot or accessed by other query engines such as Trino, Presto, Hive, or Spark. This event data can be utilized by monitoring and metric dashboards to track Pinot's activity.

Requesting comments/opinions/thoughts on this feature request from Pinot OpenSource Community

[anilsomisetty]

@xiangfu0 could you please post your thoughts on my above feature request ? I have seen your pr for supporting QueryLog System table: https://github.com/apache/pinot/pull/17169, I would like to point out a possible issue: if a broker goes down, all event data from that broker may become inaccessible or lost. However, capturing and storing event data in DeepStore would ensure the data is permanently saved.

[xiangfu0]

@xiangfu0 could you please post your thoughts on my above feature request ? I have seen your pr for supporting QueryLog System table: #17169, I would like to point out a possible issue: if a broker goes down, all event data from that broker may become inaccessible or lost. However, capturing and storing event data in DeepStore would ensure the data is permanently saved.

+1, that PR is more like a UI facing feature, not a fully prod grade feature.

[xiangfu0]

+1 for this feature. Here is my high level thoughts:

1. create BrokerQueryEventListener interface for query lifecycle:

  /** Invoked when the broker receives the query request */
  void onQuerySubmitted(QueryContext queryContext, long requestTimestampMs);

  /** Invoked when the broker receives responses from servers */
  void onQueryResponseReceived(QueryContext queryContext,
                               BrokerResponse brokerResponse,
                               long responseTimestampMs);

  /** Invoked when the full query lifecycle completes successfully */
  void onQueryCompleted(QueryEventRecord eventRecord);

  /** Invoked when an error occurs during query execution */
  void onQueryFailed(QueryEventRecord eventRecord, Throwable error);

2. QueryEventRecord(mainly coming from query planner and response), this need to be reviewed/reworked carefully for compliance:

{
  "version": 1,
  "fields": {
    "timestamp": "long",
    "queryId": "string",
    "tableName": "string",
    "tenant": "string",

    "query": "string",
    "brokerId": "string",
    "clientIp": "string",

    "totalTimeMs": "int",
    "brokerTimeMs": "int",
    "serverTimeMs": "int",
    "numServersQueried": "int",
    "numServersResponded": "int",

    "segmentsScanned": "long",
    "docsScanned": "long",
    "entriesScannedInFilter": "long",
    "entriesScannedPostFilter": "long",

    "exceptions": {
      "type": "array",
      "items": "string"
    },

    "multistage": {
      "type": "map",
      "values": "string"
    },

    "metadata": {
      "type": "map",
      "values": "string"
    }
  }
}

3. write your implementation for the query logs persistency like Parquet file on deep store etc.

[anilsomisetty]

@xiangfu0 Thank you, I will work on this feature and share a design doc for review.