apache
Allow submiting ingestionJobSpec yaml to launch minion task for ingestion job
Currently, users run this command to ingest data: pinot-admin.sh LaunchIngestionJob -jobSpec my-ingestion-job-spec.yaml.
The goal is to make users can re-use the same yaml for segment ingestion in minion.
Steps:
- Add an endpoint in the Pinot controller to allow users to POST the yaml file
- read yaml to SegmentGenerationJobSpec, launch minion tasks
- Return the tasks generated map.
For simplicity, if this minion task can honor all the configs from SegmentGenerationJobSpec, then we can just convert SegmentGenerationJobSpec to a SegmentGenerationAndPushTask Minion task.
And if not, we need to wrap those SegmentGenerationJobRunner and SegmentMetadataPushJobRunner jobs and make them minon tasks.
Ref: org.apache.pinot.core.query.executor.sql.SqlQueryExecutor : how Insert INTO statement is executed. org.apache.pinot.plugin.minion.tasks.segmentgenerationandpush.SegmentGenerationAndPushTaskGenerator: how the underlying minion task generator and executor.
