amoro icon indicating copy to clipboard operation
amoro copied to clipboard

[Feature]: Unify Changestore and Logstore for flink source

Open zhoujinsong opened this issue 3 years ago • 0 comments

Description

Unify Changestore and Logstore for flink source

Use case/motivation

Arctic use Changestore and Logstore to store CDC messages. Logstore use MQ(eg: Kafka) to distribute real-time cdc message. Changestore use iceberg table to store cdc message for analyse and provide near real-time incremental pull. In really use case, Logstore store message for several days and Changestore store message for a longer time. Flink source should support to read Changestore for early CDC messages and change to Logstore when catch it up.

Describe the solution

  • Support to config the reading start time for Flink source
  • Flink source determin read Changestore or Logstore according reading start time
  • Flink source should transfer from Changestore to Logstore when catch it up.

Related issues

No response

Are you willing to submit a PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

zhoujinsong avatar Aug 31 '22 07:08 zhoujinsong