doris icon indicating copy to clipboard operation
doris copied to clipboard

[Enhencement](Jdbc catalog) Add two optional properties for jdbc catalog

Open BePPPower opened this issue 1 year ago • 2 comments

Proposed changes

Issue Number: close #xxx

Problem summary

  1. The first property is only_specified_database: In the past, Jdbc Catalog will synchronize all database from source database. Now we add a parameter called only_specified_database to jdbc catalog to allow only the specified database to be synchronized, eg:
create resource if not exists ${resource_name} properties(
    "type"="jdbc",
    "user"="root",
    "password"="123456",
    "jdbc_url" = "jdbc:mysql://172.18.0.1:${mysql_port}/doris_test?useSSL=false",
    "driver_url" = "https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar",
    "driver_class" = "com.mysql.cj.jdbc.Driver",
    "only_specified_database" = "true"
);

if only_specified_database is true, jdbc catalog will only synchronize the database which is specified in jdbc_url.

  1. The second property is lower_case_table_names: This property will synchronize jdbc external data source table names in lower case.
create resource if not exists ${resource_name} properties(
  "type"="jdbc",
  "user"="doris_test",
  "password"="123456",
  "jdbc_url" = "jdbc:oracle:thin:@172.18.0.1:${oracle_port}:${SID}",
  "driver_url" = "https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/ojdbc8.jar",
  "driver_class" = "oracle.jdbc.driver.OracleDriver",
  "lower_case_table_names" = "true"
);

Checklist(Required)

  • [ ] Does it affect the original behavior
  • [ ] Has unit tests been added
  • [ ] Has document been added or modified
  • [ ] Does it need to update dependencies
  • [ ] Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

BePPPower avatar Feb 28 '23 11:02 BePPPower

run buildall

BePPPower avatar Mar 01 '23 15:03 BePPPower

TeamCity pipeline, clickbench performance test result: the sum of best hot time: 33.52 seconds stream load tsv: 481 seconds loaded 74807831229 Bytes, about 148 MB/s stream load json: 37 seconds loaded 2358488459 Bytes, about 60 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 27 seconds loaded 861443392 Bytes, about 30 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230301153024_clickbench_pr_106608.html

hello-stephen avatar Mar 01 '23 15:03 hello-stephen

run feut

BePPPower avatar Mar 02 '23 03:03 BePPPower