flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[FLINK-35354] Support host mapping in Flink tikv cdc

Open Mrart opened this issue 1 year ago • 5 comments

In tidb production environment deployment, there are usually two kinds of network: internal network and public network. When we use pd mode in tikv, we need to do network mapping, such as spark.tispark.host_mapping in https://github.com/pingcap/tispark/blob/master/docs/userguide_3.0.md. So I think we need support host_mapping in our Flink tikv cdc connector.

Mrart avatar May 21 '24 01:05 Mrart

I think we can specific a config prefix pass config to tikv client like cdc source connector pass config to debezium by debezium.*. This way will be flexible. @leonardBang @Mrart cc

GOODBOY008 avatar May 22 '24 07:05 GOODBOY008

I think we can specific a config prefix pass config to tikv client like cdc source connector pass config to debezium by debezium.*. This way will be flexible. @leonardBang @Mrart cc

tikv.* is configured on tikv config. hostmapping and pd config are similar to the database host configuration, and the current configuration looks more appropriate

Mrart avatar May 22 '24 13:05 Mrart

As discuss offline with @Mrart , due to tikvclient's config hostMapping has different logic with others. So the solution in this PR, I think ok.

GOODBOY008 avatar May 26 '24 12:05 GOODBOY008

I think we can specific a config prefix pass config to tikv client like cdc source connector pass config to debezium by debezium.*. This way will be flexible. @leonardBang @Mrart cc

tikv.* is configured on tikv config. hostmapping and pd config are similar to the database host configuration, and the current configuration looks more appropriate

Thanks for the help review.

Mrart avatar May 26 '24 12:05 Mrart

@leonardBang Can you help me review it again?

Mrart avatar Jun 05 '24 01:06 Mrart

@Mrart Can you rebase the branch?

GOODBOY008 avatar Jul 09 '24 05:07 GOODBOY008