[Bug] [connector-v2] Dependency Conflict
Search before asking
- [X] I had searched in the issues and found no similar issues.
What happened
The jar packages in the lib directory have dependency conflicts. For example, when running the s3 plugin, there are multiple Hadoop dependent jar files (hive jar files) that cause errors. When uploading other JDBC drivers, running the es plugin results in HTTP dependency jar conflicts. How to resolve such dependency conflicts
SeaTunnel Version
dev,2.3.8
SeaTunnel Config
{
"env": {
"execution.parallelism": "1",
"job.mode": "BATCH"
},
"source": [
{
"plugin_name":"jdbc",
"url":"jdbc:postgresql://******:5432/oceandatum?currentSchema=dw_label",
"password":"******",
"user":"******",
"driver":"org.postgresql.Driver",
"query": "SELECT \"id\",\"name\",\"label_column_fact\" FROM \"label_group_a\"",
"result_table_name": "fake"
}
],
"transform": [
{"plugin_name": "sql",
"source_table_name": "fake",
"result_table_name": "fake2",
"query": "SELECT concat(id,'$test') AS id,name AS label,concat('{','\"id\":\"',id,'\",','\"name\":\"',name,'\",','\"label_column_fact\":\"',label_column_fact,'\"','}') AS property FROM fake "
}
],
"sink": [
{
"plugin_name": "elasticsearch",
"hosts":["******:9200"],
"index":"relation",
"primary_keys":["id"]
}
]
}
Running Command
bin/seatunnel.sh -c demo.json
Error Exception
Caused by: java.io.IOException: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider instantiation exception: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
Zeta or Flink or Spark Version
No response
Java or Scala Version
1.8
Screenshots
Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Reason for S3 plugin error: The guava version in the Hive driver is lower, while the guava version in the S3 plugin is higher, but now the driver dependencies are all in lib. How to eliminate this dependency conflict problem
该行与hadoop-aws包也有冲突。
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
I fix the problem.