[Discuss][CheckPoint Storage & IMap Storage] Support use JDBC to storage the checkpoint state data and IMap data

[EricJoy2048]

Search before asking

• [X] I had searched in the feature and found no similar feature requirement.

Description

Now, If you want to use SeaTunnel Zeta Cluster, you must storage checkpoint state data and IMap data to distributed file system(For example, HDFS, OSS, S3). But for some enterprises, they do not have their own big data systems, HDFS, and cloud storage. Should SeaTunnel support use JDBC to storage the checkpoint state data and IMap data? I think in the case of small cluster size, use MySQL to storage checkpoint state data and IMap data can work fine.

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[hailin0]

I think in the case of small cluster size, use MySQL to storage checkpoint state data and IMap data can work fine.

+1

[CalvinKirs]

+1 db is a good choice when the amount of data is small.

[liugddx]

#3792

[liugddx]

Maybe there are multiple options available, such as mongodb.

[2000liux]

This is a good proposal。

• For example, use rocksdb, an embedded file database. Lightweight and easy to use.

• The experience of rocksdb can be learned from flink

• if useing database 、We need to consider the high availability of the database

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.