[Feature][spark-connector-jdbc] add pre sql and post sql #1386

[zhaomin1423]

Purpose of this pull request

add pre sql and post sql #1386

Check list

• [ ] Code changed are covered with tests, or it does not need tests for reason:
• [x] If any new Jar binary package adding in you PR, please add License Notice according New License Guide
• [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

[zhaomin1423]

Hi，Could you like to help me review it? Also, the license checker were not successful. How should I do? @CalvinKirs @asdf2014 @wuchunfu

[CalvinKirs]

I'll check it out later, there is a problem with the license check, but you don‘t added a new Jar, you can ignore this.

[zhaomin1423]

I'll check it out later, there is a problem with the license check, but you don‘t added a new Jar, you can ignore this.

Thanks.

[CalvinKirs]

But you need to fix the code style of your code, we have scala-style plugin, you can run it locally

[zhaomin1423]

The pre sqls and post sqls are executed in driver, the output is executed in executors. They can not use a connection. The post sqls use a new connection because the output can be running a long time, a new connection is better for using.

| | @.*** | | @.*** |

---- 回复的原邮件 ---- | 发件人 | @.> | | 日期 | 2022年03月04日 08:57 | | 收件人 | @.> | | 抄送至 | Xiao @.@.> | | 主题 | Re: [apache/incubator-seatunnel] [Feature][spark-connector-jdbc] add pre sql and post sql #1386 (PR #1390) |

@yx91490 commented on this pull request.

In seatunnel-connectors/seatunnel-connector-spark-jdbc/src/main/scala/org/apache/seatunnel/spark/JdbcConfigs.scala:

• • Licensed to the Apache Software Foundation (ASF) under one or more

• • contributor license agreements. See the NOTICE file distributed with
• • this work for additional information regarding copyright ownership.
• • The ASF licenses this file to You under the Apache License, Version 2.0
• • (the "License"); you may not use this file except in compliance with
• • the License. You may obtain a copy of the License at
• • http://www.apache.org/licenses/LICENSE-2.0
• • Unless required by applicable law or agreed to in writing, software
• • distributed under the License is distributed on an "AS IS" BASIS,
• • WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
• • See the License for the specific language governing permissions and
• • limitations under the License.
• */ +package org.apache.seatunnel.spark

is JdbcConfigs.scala will be shared by source and sink? if not, maybe can move to org.apache.seatunnel.spark.sink package.

In seatunnel-connectors/seatunnel-connector-spark-jdbc/src/main/scala/org/apache/seatunnel/spark/sink/Jdbc.scala:

class Jdbc extends SparkBatchSink {

• private val LOG: Logger = LoggerFactory.getLogger(classOf[Jdbc])
• private var jdbcConfigs: JdbcConfigs = _
• private def getConnection(): Connection = {
• Class.forName(jdbcConfigs.driver)
• val connection = DriverManager.getConnection(jdbcConfigs.url, jdbcConfigs.user, jdbcConfigs.password)

It seems that connection will be created about 3 times at worst, can connection be reused in the same job?

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.Message ID: @.***>

[zhaomin1423]

But you need to fix the code style of your code, we have scala-style plugin, you can run it locally

Where is the checkStyle result? I am confused.

[wuchunfu]

But you need to fix the code style of your code, we have scala-style plugin, you can run it locally

Where is the checkStyle result? I am confused.

@zhaomin1423 I think you can implement scalastyle in idea

[zhaomin1423]

But you need to fix the code style of your code, we have scala-style plugin, you can run it locally

Where is the checkStyle result? I am confused.

@zhaomin1423 I think you can implement scalastyle in idea

Thanks, I found them.

[zhaomin1423]

Hi, @CalvinKirs @wuchunfu @asdf2014 Could you like to help me review it. I will do more work based on it.

[asdf2014]

Hi @zhaomin1423, would you please merge the dev branch into this PR to pass the CI, thanks

[zhaomin1423]

Hi @zhaomin1423, would you please merge the dev branch into this PR to pass the CI, thanks

Thanks, I done it.

[asdf2014]

@zhaomin1423 You are welcome

[lppsuixn]

How is this going? i need it~thanks @asdf2014

[EricJoy2048]

Hi @zhaomin1423 This feature is a very good contribution. The Connector V2 JDBC need this feature too. Can you complete it and add it to Connector V2 JDBC too？

[EricJoy2048]

This Pr is closed because it has not received a response for too long