seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

[Improve] [SourceConnector] config supports only specific fields

Open xiaofan2022 opened this issue 3 years ago • 0 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

config should be able to set all type fields for the corresponding component

SeaTunnel Version

2.1.3

SeaTunnel Config

env {
  spark.app.name = "SeaTunnel"
    spark.executor.instances = 2
    spark.executor.cores = 1
    spark.executor.memory = "1g"
    spark.master = local
}

source {
 hive {
     pre_sql = "SELECT user_id, fans_count FROM temp.xhs_user_portrait_count  limit 10"
     result_table_name = "t_table"
   }
}
transform {
}

sink {
    jdbc {
        saveMode = "Overwrite",
        truncate="true",
        driver = "com.mysql.cj.jdbc.Driver",
        url = "jdbc:mysql:///test?rewriteBatchedStatements=true",
        isolationLevel="NONE",
        user = "",
        password = "",
        dbTable = "t_test",
        customUpdateStmt = "INSERT INTO t_test (user_id,fans_count) values(?,?)"
        jdbc.connect_timeout = 10000
        jdbc.socket_timeout = 10000
      }
}

Running Command

./bin/start-seatunnel-spark.sh --master yarn --deploy-mode client --config ./test.conf

Error Exception

The jdbc sink component does not support truncate, causing the drop table

Flink or Spark Version

truncate is not supported and the setting causes drop table

Java or Scala Version

1.8

Screenshots

No response

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

xiaofan2022 avatar Dec 29 '22 07:12 xiaofan2022