seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

[Bug] [connector-v2] Dependency Conflict

Open Xuzhengz opened this issue 1 year ago • 3 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

The jar packages in the lib directory have dependency conflicts. For example, when running the s3 plugin, there are multiple Hadoop dependent jar files (hive jar files) that cause errors. When uploading other JDBC drivers, running the es plugin results in HTTP dependency jar conflicts. How to resolve such dependency conflicts

SeaTunnel Version

dev,2.3.8

SeaTunnel Config

{
  "env": {
    "execution.parallelism": "1",
    "job.mode": "BATCH"
  },
  "source": [
     {
      "plugin_name":"jdbc",
      "url":"jdbc:postgresql://******:5432/oceandatum?currentSchema=dw_label",
      "password":"******",
      "user":"******",
      "driver":"org.postgresql.Driver",
      "query": "SELECT \"id\",\"name\",\"label_column_fact\" FROM \"label_group_a\"",
      "result_table_name": "fake"
    }
  ],
  "transform": [
    {"plugin_name": "sql",
      "source_table_name": "fake",
      "result_table_name": "fake2",
      "query": "SELECT  concat(id,'$test') AS id,name AS label,concat('{','\"id\":\"',id,'\",','\"name\":\"',name,'\",','\"label_column_fact\":\"',label_column_fact,'\"','}') AS property FROM fake "
}
  ],
  "sink": [
    {
      "plugin_name": "elasticsearch",
      "hosts":["******:9200"],
      "index":"relation",
      "primary_keys":["id"]
    }
  ]
}

Running Command

bin/seatunnel.sh -c demo.json

Error Exception

Caused by: java.io.IOException: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider instantiation exception: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V

Zeta or Flink or Spark Version

No response

Java or Scala Version

1.8

Screenshots

image

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

Xuzhengz avatar Aug 30 '24 02:08 Xuzhengz

Reason for S3 plugin error: The guava version in the Hive driver is lower, while the guava version in the S3 plugin is higher, but now the driver dependencies are all in lib. How to eliminate this dependency conflict problem

Xuzhengz avatar Aug 30 '24 03:08 Xuzhengz

image 该行与hadoop-aws包也有冲突。

zhangqing123 avatar Sep 02 '24 09:09 zhangqing123

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] avatar Oct 04 '24 00:10 github-actions[bot]

I fix the problem.

dailai avatar Nov 06 '24 06:11 dailai