seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

[Feature][Connector-V2][Milvus] Support Milvus source & sink

Open Thomas-HuWei opened this issue 1 year ago • 12 comments

Purpose of this pull request

Support Milvus source & sink.

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

  • [ ] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
  • [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
  • [x] If you are contributing the connector code, please check that the following files are updated:
    1. Update change log that in connector document. For more details you can refer to connector-v2
    2. Update plugin-mapping.properties and add new connector information in it
    3. Update the pom file of seatunnel-dist
  • [ ] Update the release-note.

Thomas-HuWei avatar Jun 25 '24 09:06 Thomas-HuWei

cc @liugddx as well

Hisoka-X avatar Jun 25 '24 09:06 Hisoka-X

Hi @Thomas-HuWei , thanks for your contribute! Please follow the guide to open github action on your fork repository. https://github.com/apache/seatunnel/pull/7060/checks?check_run_id=26644733039

Hisoka-X avatar Jun 26 '24 02:06 Hisoka-X

https://github.com/apache/seatunnel/pull/7060/checks?check_run_id=26644733039

image

I've done. Next what should I do?

Thomas-HuWei avatar Jun 26 '24 11:06 Thomas-HuWei

https://github.com/apache/seatunnel/pull/7060/checks?check_run_id=26644733039

image

I've done. Next what should I do?

push an emtpy commit.

Hisoka-X avatar Jun 26 '24 11:06 Hisoka-X

Please format the code by executing mvn spotless:apply

liugddx avatar Jun 26 '24 11:06 liugddx

Please format the code by executing mvn spotless:apply

I've do spotless, but it seems that docs can't pass the ci. And do you have any contact ways? I want to chat something with you about seatunnel support vector database.

Thomas-HuWei avatar Jun 27 '24 07:06 Thomas-HuWei

Please format the code by executing mvn spotless:apply

I've do spotless, but it seems that docs can't pass the ci. And do you have any contact ways? I want to chat something with you about seatunnel support vector database.

Our operations staff has WeChat, and you can join the WeChat group through him. wechat id seatunnel1

Hisoka-X avatar Jun 27 '24 08:06 Hisoka-X

Please fix ci.

Hisoka-X avatar Jul 01 '24 03:07 Hisoka-X

Please fix ci.

Why spotless check Milvus.md? image

How can I fix it? Delete these docs?

Thomas-HuWei avatar Jul 01 '24 03:07 Thomas-HuWei

Please fix ci.

Why spotless check Milvus.md? image

How can I fix it? Delete these docs?

Please use mvn spotless:apply to format doc before commit.

Hisoka-X avatar Jul 01 '24 04:07 Hisoka-X

@Hisoka-X I had fix my code ci problem, but there still exist ci error.

Thomas-HuWei avatar Jul 01 '24 11:07 Thomas-HuWei

@Hisoka-X I had fix my code ci problem, but there still exist ci error.

Please retry the failed ci image

Hisoka-X avatar Jul 01 '24 11:07 Hisoka-X

@Hisoka-X These two job, I have retry many times, but it still fail. image

image

Thomas-HuWei avatar Jul 02 '24 06:07 Thomas-HuWei

Seem like the engine-k8s-it failed reason not related with this PR. Never mind.

Hisoka-X avatar Jul 02 '24 07:07 Hisoka-X

Seem like the engine-k8s-it failed reason not related with this PR. Never mind.

Fixed by #7101

Hisoka-X avatar Jul 04 '24 02:07 Hisoka-X

@Hisoka-X @liugddx I have add e2e test, but it runs fail. image

Do you know how to solve this problem?

Thomas-HuWei avatar Jul 04 '24 11:07 Thomas-HuWei

Implement SupportMultiTableSink to support concurrent writes

https://github.com/apache/seatunnel/blob/57e5627e89e870f26c1b0774cc225eaabe7f3c9a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSink.java#L63

https://github.com/apache/seatunnel/blob/57e5627e89e870f26c1b0774cc225eaabe7f3c9a/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcSinkWriter.java#L46

https://github.com/apache/seatunnel/blob/57e5627e89e870f26c1b0774cc225eaabe7f3c9a/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/SinkCommonOptions.java#L27

MultiTable metadata placeholders

https://github.com/apache/seatunnel/blob/57e5627e89e870f26c1b0774cc225eaabe7f3c9a/seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/sink/PaimonSinkFactory.java#L77

hailin0 avatar Jul 10 '24 09:07 hailin0

please link to https://github.com/apache/seatunnel/pull/7158

Thomas-HuWei avatar Jul 10 '24 12:07 Thomas-HuWei