[BackPort for 0.11.0 Release] Add Protocol Buffer Stream Decoder

[KKcorps]

trafficstars

(cherry picked from commit 8806dc35ecd09297c72d987b80f37b190dd7974e) Original PR - #8972

Supported configs

• descriptorFile - Path of the descriptor file. You can generate this file use protoc -o file.desc --include_imports file.proto command. The path can be a local path in which case it needs to be available on all servers OR it can be a DFS path such as s3, gcs etc. If providing a DFS path, you need to ensure pinot is configured to use that filesystem.

• protoClassName - If the descriptor file contains multiple proto object, you can mention the name of the class to use for parsing.

Example

    "streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.consumer.type": "lowLevel",
        "stream.kafka.topic.name": "metrics_pb",
        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.ProtoBufMessageDecoder",
        "stream.kafka.decoder.prop.descriptorFile": "file:///tmp/Workspace/protobuf/metrics.desc",
        "stream.kafka.decoder.prop.protoClassName": "Metrics",
        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.broker.list": "localhost:19092",
        "stream.kafka.consumer.prop.auto.offset.reset": "largest",
        "realtime.segment.flush.threshold.time": "12h",
        "realtime.segment.flush.threshold.size": "100M"
      }

[walterddr]

let's wait for #9176