flink-connector-jdbc [FLINK-32068] connector jdbc support clickhouse

the clickhouse in batch mode ,as sink and source
the clickhouse in streaming mode ,as sink

May 18 '23 15:05 WenDing-Y

Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)

May 18 '23 15:05 boring-cyborg[bot]

@WenDing-Y Could you please rebase?

May 24 '23 10:05 MartijnVisser

Okay, I have rebased, but I found that there were quite a few modifications in the testing section. I have already submitted some code, but the unit test cannot pass now. I am still modifying this section. @MartijnVisser

May 24 '23 17:05 WenDing-Y

the pr is ready

May 25 '23 13:05 WenDing-Y

Please help me review, thank you very much @snuyanzin

May 29 '23 12:05 WenDing-Y

compile_and_test (1.18-SNAPSHOT) fail, It doesn't seem like the problem was caused by my submission

May 31 '23 13:05 WenDing-Y

Please help me review, where else can I improve? Thank you very much @eskabetxe

May 31 '23 14:05 WenDing-Y

What other tasks are needed for this PR to merge

Jun 02 '23 03:06 WenDing-Y

What other tasks are needed for this PR to merge

You will need to rebase, because there are a couple of merge commits when you merged main into your branch. Those merge commits prevent merging; by rebasing, those commits should not be needed anymore

Jun 05 '23 06:06 MartijnVisser

Ok, I have rebase @MartijnVisser

Jun 05 '23 13:06 WenDing-Y

Ok, I have rebase @MartijnVisser

I don't think you have, given that there are 4 merge-commits in this PR which shouldn't be there when rebased properly: bb13e1f51d071c5180ccf7d658839a3afac61468 bced473fbb8a606dca287fcf9329c7f052b9c4be 9abdfafac76b17dcce371239ff5e16d9b90b9980 2655b6badefbea699ddb2465803076ec229b884b

Jun 05 '23 13:06 MartijnVisser

Sorry i was late here, thanks for your PR @WenDing-Y The thing that I do not understand... why do we need 2 ClickhouseDialect classes

flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/clickhouse/ClickhouseDialect.java
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/databases/clickhouse/dialect/ClickHouseDialect.java

and 2 ClickhouseDialectFactory

flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/databases/clickhouse/dialect/ClickHouseDialectFactory.java
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/clickhouse/ClickhouseDialectFactory.java

Jun 05 '23 13:06 snuyanzin

Ok, I have rebase @MartijnVisser

I don't think you have, given that there are 4 merge-commits in this PR which shouldn't be there when rebased properly: bb13e1f bced473 9abdfaf 2655b6b

Sorry, it's my problem. I have rebase it again @MartijnVisser

Jun 05 '23 16:06 WenDing-Y

Sorry i was late here, thanks for your PR @WenDing-Y The thing that I do not understand... why do we need 2 ClickhouseDialect classes

flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/clickhouse/ClickhouseDialect.java
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/databases/clickhouse/dialect/ClickHouseDialect.java

and 2 ClickhouseDialectFactory

flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/databases/clickhouse/dialect/ClickHouseDialectFactory.java
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/clickhouse/ClickhouseDialectFactory.java

Sorry, there was a slight error in my rebase just now, which caused this to happen. Now this has always been resolved

Jun 05 '23 16:06 WenDing-Y

Now this has always been resolved

Unfortunately it hasn't been resolved. In case you want to rebase, what you should do is the following (this is assuming that you've added this repository as a remote called upstream.)

Run git fetch upstream
Run git rebase upstream/main and complete this until the rebase has successfully completed
Force push your changes to your branch

Jun 05 '23 17:06 MartijnVisser

Now this has always been resolved

Unfortunately it hasn't been resolved. In case you want to rebase, what you should do is the following (this is assuming that you've added this repository as a remote called upstream.)

Run git fetch upstream

Run git rebase upstream/main and complete this until the rebase has successfully completed

Force push your changes to your branch

It seems like this, my local submit history, the new feature started working after [Flink-31551] @MartijnVisser

Jun 06 '23 05:06 WenDing-Y

if you execute this command on your branch

git log --pretty-oneline

you'll se several merge remote-tracking commits like

c9e316596dc29fe3cb60578f53dce3b2b7fa8c1a (HEAD, WenDing/jdbc-clickhouse) Merge remote-tracking branch 'origin/jdbc-clickhouse' into jdbc-clickhouse
9098ce56b0d703cda0a1f24f69058b9ab15af556 [FLINK-32068]connector jdbc support clickhouse
8f6eff4e98003d568da8e722c06167dd491d4b53 [FLINK-32068]connector jdbc support clickhouse
79cc7987be514afd0580d377208c798ec04a9415 [FLINK-32068] jdbc support clickhouse
210968dca5a1fb47cd3268fa130f0ff5e54f299c [FLINK-32068]connector jdbc support clickhouse
c49485360a58e12ce7aad13ee627dcecc74c891e [FLINK-32068] connector jdbc support clickhouse
00c94f815e8fda71777a83d63bd1e14111bc5650 [FLINK-32068] connector jdbc support clickhouse
cd268a2111ecfd0602f4150d5b1cc44b99779456 [FLINK-32068] connector jdbc support clickhouse
86b5eeee7b88c1ef46f7d768875762ae253eec97 [FLINK-32068] connector jdbc support clickhouse
b81bc533fb98da2e5a76bddb22a4da8a52fb411e [FLINK-32068]connector jdbc support clickhouse
55f79f215456c0dbb5fdd3d99a8be6985be092ef [FLINK-32068]connector jdbc support clickhouse
1ff9e70dfbeefaa2e92c11db36fcca3c27bf9b49 Merge remote-tracking branch 'origin/jdbc-clickhouse' into jdbc-clickhouse
b6663a5fced2e4703c263827ca67c2dbacbf96bd [FLINK-32068]connector jdbc support clickhouse
2655b6badefbea699ddb2465803076ec229b884b Merge remote-tracking branch 'origin/main' into jdbc-clickhouse
9abdfafac76b17dcce371239ff5e16d9b90b9980 Merge branch 'main' into jdbc-clickhouse
d67de5f0f94b1de912470028ea1fc74376bd64b8 [FLINK-31551] Add support for CrateDB (#29)
7b8617e03048d32d44dc453bd556d9f59d58a3e1 [FLINK-32068]connector jdbc support clickhouse,delete testcontainers version
bced473fbb8a606dca287fcf9329c7f052b9c4be Merge remote-tracking branch 'upstream/main' into jdbc-clickhouse

and this the issue

Jun 06 '23 05:06 snuyanzin

if you execute this command on your branch

git log --pretty-oneline

you'll se several merge remote-tracking commits like

c9e316596dc29fe3cb60578f53dce3b2b7fa8c1a (HEAD, WenDing/jdbc-clickhouse) Merge remote-tracking branch 'origin/jdbc-clickhouse' into jdbc-clickhouse
9098ce56b0d703cda0a1f24f69058b9ab15af556 [FLINK-32068]connector jdbc support clickhouse
8f6eff4e98003d568da8e722c06167dd491d4b53 [FLINK-32068]connector jdbc support clickhouse
79cc7987be514afd0580d377208c798ec04a9415 [FLINK-32068] jdbc support clickhouse
210968dca5a1fb47cd3268fa130f0ff5e54f299c [FLINK-32068]connector jdbc support clickhouse
c49485360a58e12ce7aad13ee627dcecc74c891e [FLINK-32068] connector jdbc support clickhouse
00c94f815e8fda71777a83d63bd1e14111bc5650 [FLINK-32068] connector jdbc support clickhouse
cd268a2111ecfd0602f4150d5b1cc44b99779456 [FLINK-32068] connector jdbc support clickhouse
86b5eeee7b88c1ef46f7d768875762ae253eec97 [FLINK-32068] connector jdbc support clickhouse
b81bc533fb98da2e5a76bddb22a4da8a52fb411e [FLINK-32068]connector jdbc support clickhouse
55f79f215456c0dbb5fdd3d99a8be6985be092ef [FLINK-32068]connector jdbc support clickhouse
1ff9e70dfbeefaa2e92c11db36fcca3c27bf9b49 Merge remote-tracking branch 'origin/jdbc-clickhouse' into jdbc-clickhouse
b6663a5fced2e4703c263827ca67c2dbacbf96bd [FLINK-32068]connector jdbc support clickhouse
2655b6badefbea699ddb2465803076ec229b884b Merge remote-tracking branch 'origin/main' into jdbc-clickhouse
9abdfafac76b17dcce371239ff5e16d9b90b9980 Merge branch 'main' into jdbc-clickhouse
d67de5f0f94b1de912470028ea1fc74376bd64b8 [FLINK-31551] Add support for CrateDB (#29)
7b8617e03048d32d44dc453bd556d9f59d58a3e1 [FLINK-32068]connector jdbc support clickhouse,delete testcontainers version
bced473fbb8a606dca287fcf9329c7f052b9c4be Merge remote-tracking branch 'upstream/main' into jdbc-clickhouse

and this the issue

What else do I need to do? I executed it 1.Run git fetch upstream 2.Run git rebase upstream/main and complete this until the rebase has successfully completed 3.Force push your changes to your branch

Jun 06 '23 06:06 WenDing-Y

how do you recognize that rebase has successfully completed? I tried to do it with your branch (git rebase upstream/main) and got a number of conflicts

Jun 06 '23 07:06 snuyanzin

I have identified the problem and during the final step, I executed a push instead of a force push, which caused the git rebase to fall into a loop. Thank you for your guidance @MartijnVisser @snuyanzin

Jun 06 '23 15:06 WenDing-Y

how do you recognize that rebase has successfully completed? I tried to do it with your branch (git rebase upstream/main) and got a number of conflicts

The problem should be solved, you can try again

Jun 06 '23 15:06 WenDing-Y

What other tasks are needed for this PR to merge

You will need to rebase, because there are a couple of merge commits when you merged main into your branch. Those merge commits prevent merging; by rebasing, those commits should not be needed anymore

the pr is ready @MartijnVisser

Jun 07 '23 08:06 WenDing-Y

@MartijnVisser the pr is ready

Jun 13 '23 02:06 WenDing-Y

What other tasks are needed for this PR to merge @MartijnVisser

Jun 15 '23 06:06 WenDing-Y

What other tasks are needed for this PR to merge

Please mark review items as resolved, if you really have resolved them. Right now, it looks like there are still open items.

Jun 15 '23 12:06 MartijnVisser

Can you provide more specific information, such as error messages

Jun 16 '23 11:06 WenDing-Y

One question that comes to my mind now.. This will only work with a 1 server no? if we have a Clickhouse cluster with more than 1 server (that will have a zookeeper envolved) will not work no?

if this is true, I think we should have a note on documentation

I tried to work normally in cluster mode

My steps are as follows 1.create ck table

create table if not exists default.ck_flink_test_local on cluster clicks_cluster
(
    user_id Int32,
    message String
)
engine = ReplicatedMergeTree('/clickhouse/tables/{shard}/ck_flink_test_local', '{replica}')
PRIMARY KEY (user_id)




create table if not exists default.ck_flink_test on cluster clicks_cluster
(
user_id Int32,
    message String
)
engine = Distributed('clicks_cluster', 'default', 'ck_flink_test_local', user_id);

in flink sql client


CREATE TABLE ck_flink_test(
   user_id INTEGER,
  message String
) WITH (
  'connector' = 'jdbc',
  'url' = 'jdbc:clickhouse://xx:xx/default',
 'table-name' = 'ck_flink_test',
 'username’='xx',
 'password’='xx'
);

exec insert sql

INSERT INTO ck_flink_test (user_id, message) VALUES
    (101, 'Hello');

thd data can be viewed on the ck client

select * from ck_flink_test;

5.the flink sql client

Jul 11 '23 12:07 WenDing-Y

One question that comes to my mind now.. This will only work with a 1 server no? if we have a Clickhouse cluster with more than 1 server (that will have a zookeeper envolved) will not work no? if this is true, I think we should have a note on documentation

I tried to work normally in cluster mode

My steps are as follows 1.create ck table
create table if not exists default.ck_flink_test_local on cluster clicks_cluster
(
    user_id Int32,
    message String
)
engine = ReplicatedMergeTree('/clickhouse/tables/{shard}/ck_flink_test_local', '{replica}')
PRIMARY KEY (user_id)




create table if not exists default.ck_flink_test on cluster clicks_cluster
(
user_id Int32,
    message String
)
engine = Distributed('clicks_cluster', 'default', 'ck_flink_test_local', user_id);
in flink sql client
CREATE TABLE ck_flink_test(
   user_id INTEGER,
  message String
) WITH (
  'connector' = 'jdbc',
  'url' = 'jdbc:clickhouse://xx:xx/default',
 'table-name' = 'ck_flink_test',
 'username’='xx',
 'password’='xx'
);
exec insert sql
INSERT INTO ck_flink_test (user_id, message) VALUES
    (101, 'Hello');
thd data can be viewed on the ck client
select * from ck_flink_test;
5.the flink sql client

Jul 11 '23 12:07 WenDing-Y

One question that comes to my mind now.. This will only work with a 1 server no? if we have a Clickhouse cluster with more than 1 server (that will have a zookeeper envolved) will not work no? if this is true, I think we should have a note on documentation

I tried to work normally in cluster mode

My steps are as follows 1.create ck table
create table if not exists default.ck_flink_test_local on cluster clicks_cluster
(
    user_id Int32,
    message String
)
engine = ReplicatedMergeTree('/clickhouse/tables/{shard}/ck_flink_test_local', '{replica}')
PRIMARY KEY (user_id)




create table if not exists default.ck_flink_test on cluster clicks_cluster
(
user_id Int32,
    message String
)
engine = Distributed('clicks_cluster', 'default', 'ck_flink_test_local', user_id);
in flink sql client
CREATE TABLE ck_flink_test(
   user_id INTEGER,
  message String
) WITH (
  'connector' = 'jdbc',
  'url' = 'jdbc:clickhouse://xx:xx/default',
 'table-name' = 'ck_flink_test',
 'username’='xx',
 'password’='xx'
);
exec insert sql
INSERT INTO ck_flink_test (user_id, message) VALUES
    (101, 'Hello');
thd data can be viewed on the ck client
select * from ck_flink_test;
5.the flink sql client

@eskabetxe I guess you might have defined the table incorrectly

Jul 11 '23 12:07 WenDing-Y

@WenDing-Y hello, in our project, we want to import data to clickhouse. Now this PR no longer merges with the main branch?

Nov 03 '24 13:11 Edward-Gavin