flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

Published 20 hours ago • verified publisher icon apache

[FLINK-35167][cdc-connector] Introduce MaxCompute pipeline DataSink

Open dingxin-tech opened this issue 1 year ago • 4 comments
trafficstars

dingxin-tech avatar Apr 25 '24 04:04 dingxin-tech

I am wondering how a commercial database sink like MaxCompute to do e2e test?

loserwang1024 avatar May 16 '24 06:05 loserwang1024

I am wondering how a commercial database sink like MaxCompute to do e2e test?

I will soon be working on creating a Docker image for a MaxCompute Emulator that launches a mocked version of MaxCompute. This will allow for end-to-end testing to be performed by initializing this image prior to regression testing.

dingxin-tech avatar May 21 '24 03:05 dingxin-tech

hi @loserwang1024 , I recently completed the development and release of a Docker image for the MaxCompute Emulator, and I have also added some related regression cases. Could you help me review the code in this pull request?

dingxin-tech avatar May 23 '24 12:05 dingxin-tech

Thanks @dingxin-tech for the nice work, the CI wasn't triggered properly, we need adjust the CI setting[1] when new connector or new module joining.

[1]https://github.com/apache/flink-cdc/blob/master/.github/workflows/flink_cdc.yml

Hi, I added tests for MaxCompute in the CI file. Additionally, I refactored the code to apply the newly released DataSink feature, which allows specifying a HashFunction, and upgraded the ODPS SDK. Could you please review this pr again? @leonardBang @lvyanquan

dingxin-tech avatar Jul 25 '24 05:07 dingxin-tech