Fetching latest records in Batch mode - Apache Seatunnel

[arjun-guptha]

Search before asking

• [X] I had searched in the feature and found no similar feature requirement.

Description

I have successfully deployed Seatunnel in my system. I'm currently working on synchronizing data between the source and sink in MySQL. I'm running Seatunnel in batch mode, but it's currently retrieving all records from the source and dumping them into the sink table. I need to run this job every 30 minutes, but it should only fetch the latest records from the source, not all the records.

Is there a way to configure Seatunnel to only retrieve the latest records from the source and dump them into the sink table in batch mode?

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[liugddx]

Pass the incremental value into sql every time it is scheduled. This can be solved by custom parameters

[arjun-guptha]

Hi @liugddx Thanks for your response. I'm curious to know if the current released versions of this seatunnel supports incremental offline batch synchronization ?

[liugddx]

Hi @liugddx Thanks for your response. I'm curious to know if the current released versions of this seatunnel supports incremental offline batch synchronization ?

This scenario requires the use of scheduling capabilities, DolphinScheduler might be a good choice

[baobao2016]

这个是很常用的功能，希望能添加这方面的支持

[baobao2016]

DolphinScheduler 好像也不支持动态传递sql的where条件的参数

[liugddx]

DolphinScheduler 好像也不支持动态传递sql的where条件的参数

SeaTunnel has custom parameters.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.