pinot icon indicating copy to clipboard operation
pinot copied to clipboard

Published 20 hours ago • verified publisher icon apache

Refactor to eliminate duplicate code in Spark 2.x and 3.x Batch Ingestion

Open chrajeshbabu opened this issue 4 months ago • 1 comments

Currently between spark 2.x and spark 3.x batch ingestion lot of code is duplicated other than SparkSegmentMetadataPushJobRunner class. This ticket is to refactor the code to eliminate the duplicate code without breaking the compatibility. Code changes required are to :

  • Abstract common functionality into shared methods/classes.
  • Separate version-specific details.
  • Ensure the resulting codebase is cleaner and easier to maintain.

chrajeshbabu avatar Oct 22 '24 03:10 chrajeshbabu