[rocketmq-connect] For replicator design ideas, seek advice.

[Git-Yang]

After reading the code of rocketmq connect module, I have some questions to ask for advice.

1.The current usage of rocketmq-connect is that transferred to the middle tier by sourceTask, and then transferred to the target cluster by sinkTask. For rocketmq type data sources, can they be directly synchronized to the target cluster？

2.As for RocketMQConverter, I understand whether it is a special processing for rocketmq, which should not be implemented in rocketmq-connect or implemented in RocketMQConverter. Do you have any opinions on this part of the community?

3.I plan to implement the WorkerDirctTask of direct forwarding by encapsulating WokerSourceTask and WokerSinkTask. Is this method in line with the community development plan?

[maixiaohai]

@vongosling @duhenglucky We planed to use Replicator and Connector in our company and found some design conflicts. For example, the sink part of Connect is useless in Replicator job; for order sync of Replicator, there will be some changes not only in Replicator but also in Connect. So we want some suggestions from community about the design part and future development of Connect. Thanks~

[vongosling]

The connector is a wider exchanger between any data source(including file system, event mesh, and so on). I would like to encourage you to enhance the replicator function in your case. while not implement an almost the same functionnality. Thoughts?

[maixiaohai]

Agree with you. We have some changes at Connect Project related with Replicator, we may refactor it later. At the same time, we will submit other seperate prs. Thanks for your reply in time~

[odbozhou]

After reading the code of rocketmq connect module, I have some questions to ask for advice.

1.The current usage of rocketmq-connect is that transferred to the middle tier by sourceTask, and then transferred to the target cluster by sinkTask. For rocketmq type data sources, can they be directly synchronized to the target cluster？

2.As for RocketMQConverter, I understand whether it is a special processing for rocketmq, which should not be implemented in rocketmq-connect or implemented in RocketMQConverter. Do you have any opinions on this part of the community?

3.I plan to implement the WorkerDirctTask of direct forwarding by encapsulating WokerSourceTask and WokerSinkTask. Is this method in line with the community development plan?

1. Your understanding is correct.

2. RocketMQConverter is the direct support of rocketmq source and target, and one less data transfer.

3. Of course what you said is also achievable to provide sink and source

[odbozhou]

rocketmq-connect already has an independent repository, you can go to this repository to submit PR, issue. https://github.com/apache/rocketmq-connect