seatunnel
seatunnel copied to clipboard
apache
[Umbrella][ST-Engine] Design And Task Tracking
Search before asking
- [X] I had searched in the feature and found no similar feature requirement.
Description
Discuss: #1954
This issue can help us tracking the design of SeaTunnelEngine.
- [x] SeaTunnel Server Design https://github.com/apache/incubator-seatunnel/issues/2210
- [x] SeaTunnel Client https://github.com/apache/incubator-seatunnel/issues/2260
- [x] Create LogicalPlan From JobConfigFile https://github.com/apache/incubator-seatunnel/issues/2261
- [x] Design of Submit Job to Server https://github.com/apache/incubator-seatunnel/issues/2333
- [x] Create PhysicalPlan From LogicalPlan https://github.com/apache/incubator-seatunnel/issues/2269
- [x] Design of TaskGroup Scheduler https://github.com/apache/incubator-seatunnel/issues/2339
- [x] The design of Checkpoint https://github.com/apache/incubator-seatunnel/issues/2274
- [x] The design of ExecutionService https://github.com/apache/incubator-seatunnel/issues/2279
- [x] The design of ResourceManager https://github.com/apache/incubator-seatunnel/issues/2426
- [x] How to deal with network partitions https://github.com/apache/incubator-seatunnel/issues/2430
- [x] JobMaster need support reinit when the active master is done and new master run https://github.com/apache/incubator-seatunnel/issues/2501
- [ ] Use the WAL to persistence the Job Status(Hazelcast ReplecatedMap Persistence) https://github.com/apache/incubator-seatunnel/issues/2557
- [x] https://github.com/apache/incubator-seatunnel/issues/2725
Slack Channel Link: https://apacheseatunnel.slack.com/archives/C03RQS81RQQ
Usage Scenario
No response
Related issues
No response
Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Job fault-tolerant(Checkpoint): https://github.com/apache/incubator-seatunnel/issues/2274
TaskExecutionService and Task related design : https://github.com/apache/incubator-seatunnel/issues/2279
The design proposals above are detailed designs. As a rookie, I am really interested in the following questions:
- What's the background of the new engine? What's the primary scenario? (Running on IoT device in edge computing? or substitute of flink in the cloud?)
- Since the positioning of ST is Data Integration, What's the most crucial character of the st-engine? And what is unnecessary in the st-engine but important to flink/spark (checkpoint)?
The design proposals above are detailed designs. As a rookie, I am really interested in the following questions:
- What's the background of the new engine? What's the primary scenario? (Running on IoT device in edge computing? or substitute of flink in the cloud?)
- Since the positioning of ST is Data Integration, What's the most crucial character of the st-engine? And what is unnecessary in the st-engine but important to flink/spark (checkpoint)?
@lhyundeadsoul 1: https://github.com/apache/incubator-seatunnel/issues/1954 2: Seatunnel engine will not have aggregate calculation, such as join, window and other operations that require large state
The design proposals above are detailed designs. As a rookie, I am really interested in the following questions:
- What's the background of the new engine? What's the primary scenario? (Running on IoT device in edge computing? or substitute of flink in the cloud?)
- Since the positioning of ST is Data Integration, What's the most crucial character of the st-engine? And what is unnecessary in the st-engine but important to flink/spark (checkpoint)?
@lhyundeadsoul 1: #1954 2: Seatunnel engine will not have aggregate calculation, such as join, window and other operations that require large state
Thx for your answer. No aggregate operator/ No state management, that makes sense for a data integration engine. But, Can SQL transformation plugin still be supported in st-engine?
We plan support Code Transformation first. And, In the feature, we will support SQL.
