seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

[Feature][Zeta] Replace Imap by RocksDB

Open dybyte opened this issue 2 months ago • 7 comments

Fixes: https://github.com/apache/seatunnel/issues/9851

Purpose of this pull request

This PR implements the basic RocksDB storage layer. After gathering feedback, I plan to further improve the storage code and migrate other Imap instances to RocksDB, updating the documentation along the way.

Does this PR introduce any user-facing change?

Yes. Users should now configure map-store settings in seatunnel.yaml instead of hazelcast.yaml.

How was this patch tested?

Added RocksDBServiceTest.

Check list

  • [ ] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
  • [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
  • [ ] If you are contributing the connector code, please check that the following files are updated:
    1. Update plugin-mapping.properties and add new connector information in it
    2. Update the pom file of seatunnel-dist
    3. Add ci label in label-scope-conf
    4. Add e2e testcase in seatunnel-e2e
    5. Update connector plugin_config

dybyte avatar Oct 28 '25 08:10 dybyte

Hi @Hisoka-X @zhangshenghang @liugddx, please take a look when you have time. Thanks!

dybyte avatar Oct 31 '25 23:10 dybyte

The configuration file is migrated from hazelcast.yaml to seatunnel.yaml, which is a breaking change.Can we provide a configuration migration tool or script?

zhangshenghang avatar Nov 01 '25 02:11 zhangshenghang

It is recommended to be compatible with both methods at the same time and consider being compatible with the old configuration for a period of time. In addition, since this is a major change, could you supplement your design plan?

zhangshenghang avatar Nov 01 '25 02:11 zhangshenghang

The configuration file is migrated from hazelcast.yaml to seatunnel.yaml, which is a breaking change.Can we provide a configuration migration tool or script?

It is recommended to be compatible with both methods at the same time and consider being compatible with the old configuration for a period of time. In addition, since this is a major change, could you supplement your design plan?

The design document can be found in the Apache dev mailing list. You can quickly get the context by checking the related comments there. @zhangshenghang

For compatibility, I’ll make the configuration still available in hazelcast.yaml for now. A migration tool will be provided later.

dybyte avatar Nov 02 '25 07:11 dybyte

Hi @hawk9821 , PTAL when you have a moment. Thanks!

dybyte avatar Nov 03 '25 10:11 dybyte

Just a quick reminder in case you missed this. PTAL when you have some time. Thanks! @zhangshenghang @davidzollo

dybyte avatar Nov 13 '25 09:11 dybyte

Hi @zhangshenghang . If possible, could you request a Copilot Review for this PR? I don’t have permission to request it myself, and I would appreciate any feedback.

dybyte avatar Nov 19 '25 10:11 dybyte