apache
Data ingestion CPU efficiency improvements
Pinot data ingestion from Kafka is following the 1 thread per Kafka partition mechanism. The scaling up is relying on increasing number of Kafka topic partitions. However, due to the nature of ingestion computation load, Kafka broker usually has a far higher traffic volume limit per partition than Pinot. For example, with same type of hardware, Kafka could afford traffic over 8MB/s/partition but Pinot if doing complex transformation and index building (e.g. SchemaConformingTransformer & text index) can only afford <2 MB/s/partition. This makes the Kafka partition expansion not able to be always in sync with Pinot's system load. In reality, we are observing that in a Pinot server with tens of cores, only 20% are busy with ingesting and others relatively idle.
Hence, there's requirement to improve the computation efficiency and do parallel (at least part of) single partition message processing.
From the attached pic, there are a few components to be improved:
- gzip compression -> to zstd with proper level
- transformers -> using batch and parallel processing, there are some other OSS projects like uForwarder doing the batch message processing
- indexing -> TBD
- Kafka polling -> batch polling
Good findings! Let's first check if there are some low hanging fruit optimizations. cc @swaminathanmanish
