seatunnel
seatunnel copied to clipboard
apache
[Connector-V2][JDBC-connector] support Jdbc dm
Purpose of this pull request
Check list
- [ ] Code changed are covered with tests, or it does not need tests for reason:
- [x] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
- [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
I imitate MySqlCatalog to DMCatalog, but I don't know it work. and the workflows is failed, I need some time to solve it，and I try to learn know how to do it rencently.
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
I imitate MySqlCatalog to DMCatalog, but I don't know it work. and the workflows is failed, I need some time to solve it，and I try to learn know how to do it rencently.
Ask any questions at any time.
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
ok,I will lately
I imitate MySqlCatalog to DMCatalog, but I don't know it work. and the workflows is failed, I need some time to solve it，and I try to learn know how to do it rencently.
Ask any questions at any time.
TypeMapper is in the method org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.MySqlCatalog#fromJdbcType
but also in this method org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.mysql.MySqlTypeMapper
Any difference?
another question, Mysql has the catalog named MysqlCatalog ,and why does PostgreSQL have no catalog such as PostGresqlCatalog?
I imitate MySqlCatalog to DMCatalog, but I don't know it work. and the workflows is failed, I need some time to solve it，and I try to learn know how to do it rencently.
Ask any questions at any time.
TypeMapper is in the method org.apache.seatunnel.connectors.seatunnel.jdbc.catalog.MySqlCatalog#fromJdbcType
but also in this method org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.mysql.MySqlTypeMapper
Any difference?
There is really no difference between the two, This is indeed repeated in the implementation of the catalog. However, the catalog has not been started yet, so it is not necessary to implement the catalog related things.
Refer to the implementation under the jdbc.internal.dialect package.
another question, Mysql has the catalog named MysqlCatalog ,and why does PostgreSQL have no catalog such as PostGresqlCatalog?
Because only the catalog of mysql is currently implemented, pg has not yet been implemented. DM can also temporarily not implement catalog
DM can also temporarily not implement catalog
thx，I get , I will modify my code this night
Have you tested XA-Transaction?, This will be used in exactly-once Sink. And add how to configure exactly-once sink in the document
Have you tested XA-Transaction?, This will be used in exactly-once Sink. And add how to configure exactly-once sink in the document
Thx for your guide, I will do it this weekend.
@laglangyue Wait for the JDBC e2e test (#2321) to merge. You need to add test for DM in jdbc e2e test
I have encountered a difficulty. The PostgreSQL driver is directly introduced and introduced into the Lib of connector-v2. I am afraid this is not appropriate.
In the E2E test, I need a driver in the docker. I can manually copy it to the Lib folder of connector-v2-dist, but it will fail if in the CI environment or someone else.
Therefore, I wonder whether an additional module, similar to e2e-driver-jar-dist, is required to be provided to E2E test and copied to docker.
The PostgreSQL driver is directly introduced and introduced into the Lib of connector-v2. I am afraid this is not appropriate.
@CalvinKirs Do you have any suggestions
I have 2 ideas to solve it.
-
maven-dependency-plugingenerate the dir such lib include all dirver-jar, and copy it to container, like above. - Run the command in container
wget https://repo1.maven.org/maven2/com/dameng/DmJdbcDriver18/8.1.2.141/DmJdbcDriver18-8.1.2.141.jar,and copy it to lib.
In our pom just need to provided scope @CalvinKirs
In fact, we could have a dist directory dedicated to these jars if we really needed to. (if necessary), but it seems that they are all jdbc driven, as a temporary solution, you can also use the second one. Of course, how you do it is up to you.
flink job is finish and there ara not any error log，but sink not has data
I has fix the date,time,timestamp convert in other PR，and the e2e test is finished. the dirver of DM is Apache license in maven-center-reponsitory, so I add it. such as mysql/pg, how do you think it. In my opinion, I will remove the driver from connector,and add dirver to e2e when the PR #2640 is merged. Please review it angin @hailin0 @ic4y
@ic4y @Hisoka-X @CalvinKirs I has rebased it to dev, and change dependencies,Please review again.