[ST-Engine][Design] The Design of LogicalPlan to PhysicalPlan

[Hisoka-X]

Search before asking

• [X] I had searched in the feature and found no similar feature requirement.

Description

SeaTunnel engine will receive the logical plan sent by the client, and the engine needs to convert it into a physical plan that can be directly executed. Therefore, it is necessary to process the logical execution plan and generate a physical plan through conversion. The specific process is as follows:

1. Logical Plan Received the logical plan, we need to remove redundant Actions, and verifying the Schema(Transform2 and Transform 5 should be same)
2. Execution Plan While converting to an execution plan:

• Transforms need to be merged, and the basis for merging is whether the data will be split after the Transform.
• Convert Shuffle Action to Queue
• Add Enumerator and Committer

3. Pipeline Pipeline is currently the same as ExecutionPlan. Because we didn't add the Cache module, it will be different later
4. Physical Plan We will split the Pipeline into separate executable tasks according to the degree of parallelism After this, can send task to task execution service. Then task can run normally.

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

• [x] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[2013650523]

hi, Excuse me,Let me ask you a question, Why is the data queue designed to decouple source and sink? Is the data cached in the queue when the source fails?

[Hisoka-X]

hi, Excuse me,Let me ask you a question, Why is the data queue designed to decouple source and sink? Is the data cached in the queue when the source fails?

1. The Queue will create when use shuffle transform (at now call partition transform). Used for data shuffle and change parallelism
2. The cache feature you mentioned will be added later