seatunnel [Bug] [Spark] java.nio.file.AccessDeniedException: /service/software/seatunnel/plugins.tar.gz

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

Error when running start-seatunnel-spark-connector-v2.sh

SeaTunnel Version

2.3.0-beta

SeaTunnel Config

env {
      spark.app.name = '"${jobName}"'
      spark.executor.instances = 2
      spark.executor.cores = 1
      spark.executor.memory = 1g
      spark.executor.memoryOverhead=2g
}

source {
    jdbc {
          driver = "com.mysql.jdbc.Driver"
          connection_check_timeout_sec = 100
          url = "'"${TidbOdsUrl}"'"
          user = "'"${TidbOdsUser}"'"
          password = "'"${TidbOdsPassword}"'"
          query = "select * from '${sourceBase}.${sourceTable}'"
          result_table_name = "'"${resultTable}"'"
    }
}

transform {

}

sink {
    jdbc {
          driver = "com.mysql.jdbc.Driver"
          url = "'"${DorisOdsUrl}"'"
          user = "'"${DorisOdsUser}"'"
          password = "'"${DorisOdsPassword}"'"
          query = "insert into '${sinkBase}.${sinkTable}' as select * from '${resultTable}'"
    }

}

Running Command

sh -x /service/software/seatunnel/bin/start-seatunnel-spark-connector-v2.sh --queue root.users.bzbd_batch --master yarn --deploy-mode cluster --config "${filePath}/${jobName}".conf

Error Exception

+ set -eu
	+ PRG=/service/software/seatunnel/bin/start-seatunnel-spark-connector-v2.sh
	+ '[' -h /service/software/seatunnel/bin/start-seatunnel-spark-connector-v2.sh ']'
	++ dirname /service/software/seatunnel/bin/start-seatunnel-spark-connector-v2.sh
	+ PRG_DIR=/service/software/seatunnel/bin
	++ cd /service/software/seatunnel/bin/..
	++ pwd
	+ APP_DIR=/service/software/seatunnel
	+ CONF_DIR=/service/software/seatunnel/config
	+ APP_JAR=/service/software/seatunnel/lib/seatunnel-spark-starter.jar
	+ APP_MAIN=org.apache.seatunnel.core.starter.spark.SparkStarter
	+ '[' -f /service/software/seatunnel/config/seatunnel-env.sh ']'
	+ . /service/software/seatunnel/config/seatunnel-env.sh
	++ SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
	++ FLINK_HOME=/opt/flink
[INFO] 2022-11-17 17:55:59.214  - [taskAppId=TASK-10606-32219715-46178268]:[128] -  -> ++ export SEATUNNEL_PRINT_ASCII_LOGO=true
	++ SEATUNNEL_PRINT_ASCII_LOGO=true
	+ '[' 8 == 0 ']'
	+ args='--queue root.users.bzbd_batch --master yarn --deploy-mode cluster --config /tmp/waterdrop/jobTask/tidb.conf'
	++ java -cp /service/software/seatunnel/lib/seatunnel-spark-starter.jar org.apache.seatunnel.core.starter.spark.SparkStarter --queue root.users.bzbd_batch --master yarn --deploy-mode cluster --config /tmp/waterdrop/jobTask/tidb.conf
	log4j:WARN No appenders could be found for logger (org.apache.seatunnel.core.starter.utils.CompressionUtils).
	log4j:WARN Please initialize the log4j system properly.
	log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
	Exception in thread "main" java.nio.file.AccessDeniedException: /service/software/seatunnel/plugins.tar.gz
		at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
		at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
		at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
		at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
		at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
		at java.nio.file.Files.newOutputStream(Files.java:216)
		at org.apache.seatunnel.core.starter.utils.CompressionUtils.tarGzip(CompressionUtils.java:60)
		at org.apache.seatunnel.core.starter.spark.SparkStarter$ClusterModeSparkStarter.buildCommands(SparkStarter.java:361)
		at org.apache.seatunnel.core.starter.spark.SparkStarter.main(SparkStarter.java:97)
	+ CMD=
	+ EXIT_CODE=1
	+ '[' 1 -eq 234 ']'
	+ '[' 1 -eq 0 ']'
	+ echo ''
	
	+ exit 1

Flink or Spark Version

spark

Java or Scala Version

java

Screenshots

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Nov 17 '22 10:11 wangsvip

Now I have a new problem！

The application name set in env does not take effect, it is displayed as SeaTunnel in yarn! 在env 的文件里设置app名称，在yarn 里不会显示，而是显示的是 SeaTunnel！

Master and queue must be set in env, and those set in spark-submit will not take effect！必须在env里设置 master，queue，在spark-submit 设置的不生效, 否则报！没有找到队列！

Nov 17 '22 11:11 wangsvip

The application name set in env does not take effect, it is displayed as SeaTunnel in yarn!

Answer: I will try to fix this bug.

Master and queue must be set in env, and those set in spark-submit will not take effect！

Answer:This feature is not currently supported,we will added this feature later.

Thank you for your feedback on Seatunnel.

Nov 17 '22 16:11 TaoZex

Search before asking

[x] I had searched in the issues and found no similar issues.

What happened

Error when running start-seatunnel-spark-connector-v2.sh

SeaTunnel Version

2.3.0-beta

SeaTunnel Config

env {
      spark.app.name = '"${jobName}"'
      spark.executor.instances = 2
      spark.executor.cores = 1
      spark.executor.memory = 1g
      spark.executor.memoryOverhead=2g
}

source {
    jdbc {
          driver = "com.mysql.jdbc.Driver"
          connection_check_timeout_sec = 100
          url = "'"${TidbOdsUrl}"'"
          user = "'"${TidbOdsUser}"'"
          password = "'"${TidbOdsPassword}"'"
          query = "select * from '${sourceBase}.${sourceTable}'"
          result_table_name = "'"${resultTable}"'"
    }
}

transform {

}

sink {
    jdbc {
          driver = "com.mysql.jdbc.Driver"
          url = "'"${DorisOdsUrl}"'"
          user = "'"${DorisOdsUser}"'"
          password = "'"${DorisOdsPassword}"'"
          query = "insert into '${sinkBase}.${sinkTable}' as select * from '${resultTable}'"
    }

}

Running Command

sh -x /service/software/seatunnel/bin/start-seatunnel-spark-connector-v2.sh --queue root.users.bzbd_batch --master yarn --deploy-mode cluster --config "${filePath}/${jobName}".conf

Error Exception

+ set -eu
	+ PRG=/service/software/seatunnel/bin/start-seatunnel-spark-connector-v2.sh
	+ '[' -h /service/software/seatunnel/bin/start-seatunnel-spark-connector-v2.sh ']'
	++ dirname /service/software/seatunnel/bin/start-seatunnel-spark-connector-v2.sh
	+ PRG_DIR=/service/software/seatunnel/bin
	++ cd /service/software/seatunnel/bin/..
	++ pwd
	+ APP_DIR=/service/software/seatunnel
	+ CONF_DIR=/service/software/seatunnel/config
	+ APP_JAR=/service/software/seatunnel/lib/seatunnel-spark-starter.jar
	+ APP_MAIN=org.apache.seatunnel.core.starter.spark.SparkStarter
	+ '[' -f /service/software/seatunnel/config/seatunnel-env.sh ']'
	+ . /service/software/seatunnel/config/seatunnel-env.sh
	++ SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
	++ FLINK_HOME=/opt/flink
[INFO] 2022-11-17 17:55:59.214  - [taskAppId=TASK-10606-32219715-46178268]:[128] -  -> ++ export SEATUNNEL_PRINT_ASCII_LOGO=true
	++ SEATUNNEL_PRINT_ASCII_LOGO=true
	+ '[' 8 == 0 ']'
	+ args='--queue root.users.bzbd_batch --master yarn --deploy-mode cluster --config /tmp/waterdrop/jobTask/tidb.conf'
	++ java -cp /service/software/seatunnel/lib/seatunnel-spark-starter.jar org.apache.seatunnel.core.starter.spark.SparkStarter --queue root.users.bzbd_batch --master yarn --deploy-mode cluster --config /tmp/waterdrop/jobTask/tidb.conf
	log4j:WARN No appenders could be found for logger (org.apache.seatunnel.core.starter.utils.CompressionUtils).
	log4j:WARN Please initialize the log4j system properly.
	log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
	Exception in thread "main" java.nio.file.AccessDeniedException: /service/software/seatunnel/plugins.tar.gz
		at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
		at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
		at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
		at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
		at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
		at java.nio.file.Files.newOutputStream(Files.java:216)
		at org.apache.seatunnel.core.starter.utils.CompressionUtils.tarGzip(CompressionUtils.java:60)
		at org.apache.seatunnel.core.starter.spark.SparkStarter$ClusterModeSparkStarter.buildCommands(SparkStarter.java:361)
		at org.apache.seatunnel.core.starter.spark.SparkStarter.main(SparkStarter.java:97)
	+ CMD=
	+ EXIT_CODE=1
	+ '[' 1 -eq 234 ']'
	+ '[' 1 -eq 0 ']'
	+ echo ''
	
	+ exit 1

Flink or Spark Version

spark

Java or Scala Version

java

Screenshots

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[ ] I agree to follow this project's Code of Conduct

before running first, you must be executed, as follows: tar -zcvf /service/software/seatunnel/plugins.tar.gz /service/software/seatunnel/plugins and chmod 777 /service/software/seatunnel/plugins.tar.gz

Nov 18 '22 15:11 nishuihanqiu

询问前搜索

[x] 我在问题中搜索过，没有发现类似的问题。

发生了什么

运行 start-seatunnel-spark-connector-v2.sh 时出错

海隧道版本

2.3.0-测试版

海隧道配置

env {
      spark.app.name = '"${jobName}"'
      spark.executor.instances = 2
      spark.executor.cores = 1
      spark.executor.memory = 1g
      spark.executor.memoryOverhead=2g
}

source {
    jdbc {
          driver = "com.mysql.jdbc.Driver"
          connection_check_timeout_sec = 100
          url = "'"${TidbOdsUrl}"'"
          user = "'"${TidbOdsUser}"'"
          password = "'"${TidbOdsPassword}"'"
          query = "select * from '${sourceBase}.${sourceTable}'"
          result_table_name = "'"${resultTable}"'"
    }
}

transform {

}

sink {
    jdbc {
          driver = "com.mysql.jdbc.Driver"
          url = "'"${DorisOdsUrl}"'"
          user = "'"${DorisOdsUser}"'"
          password = "'"${DorisOdsPassword}"'"
          query = "insert into '${sinkBase}.${sinkTable}' as select * from '${resultTable}'"
    }

}

运行命令

sh -x /service/software/seatunnel/bin/start-seatunnel-spark-connector-v2.sh --queue root.users.bzbd_batch --master yarn --deploy-mode cluster --config "${filePath}/${jobName}".conf

错误异常

+ set -eu
	+ PRG=/service/software/seatunnel/bin/start-seatunnel-spark-connector-v2.sh
	+ '[' -h /service/software/seatunnel/bin/start-seatunnel-spark-connector-v2.sh ']'
	++ dirname /service/software/seatunnel/bin/start-seatunnel-spark-connector-v2.sh
	+ PRG_DIR=/service/software/seatunnel/bin
	++ cd /service/software/seatunnel/bin/..
	++ pwd
	+ APP_DIR=/service/software/seatunnel
	+ CONF_DIR=/service/software/seatunnel/config
	+ APP_JAR=/service/software/seatunnel/lib/seatunnel-spark-starter.jar
	+ APP_MAIN=org.apache.seatunnel.core.starter.spark.SparkStarter
	+ '[' -f /service/software/seatunnel/config/seatunnel-env.sh ']'
	+ . /service/software/seatunnel/config/seatunnel-env.sh
	++ SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
	++ FLINK_HOME=/opt/flink
[INFO] 2022-11-17 17:55:59.214  - [taskAppId=TASK-10606-32219715-46178268]:[128] -  -> ++ export SEATUNNEL_PRINT_ASCII_LOGO=true
	++ SEATUNNEL_PRINT_ASCII_LOGO=true
	+ '[' 8 == 0 ']'
	+ args='--queue root.users.bzbd_batch --master yarn --deploy-mode cluster --config /tmp/waterdrop/jobTask/tidb.conf'
	++ java -cp /service/software/seatunnel/lib/seatunnel-spark-starter.jar org.apache.seatunnel.core.starter.spark.SparkStarter --queue root.users.bzbd_batch --master yarn --deploy-mode cluster --config /tmp/waterdrop/jobTask/tidb.conf
	log4j:WARN No appenders could be found for logger (org.apache.seatunnel.core.starter.utils.CompressionUtils).
	log4j:WARN Please initialize the log4j system properly.
	log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
	Exception in thread "main" java.nio.file.AccessDeniedException: /service/software/seatunnel/plugins.tar.gz
		at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
		at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
		at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
		at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
		at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
		at java.nio.file.Files.newOutputStream(Files.java:216)
		at org.apache.seatunnel.core.starter.utils.CompressionUtils.tarGzip(CompressionUtils.java:60)
		at org.apache.seatunnel.core.starter.spark.SparkStarter$ClusterModeSparkStarter.buildCommands(SparkStarter.java:361)
		at org.apache.seatunnel.core.starter.spark.SparkStarter.main(SparkStarter.java:97)
	+ CMD=
	+ EXIT_CODE=1
	+ '[' 1 -eq 234 ']'
	+ '[' 1 -eq 0 ']'
	+ echo ''
	
	+ exit 1

Flink 或 Spark 版本

火花

Java 或 Scala 版本

爪哇

截图

你愿意提交PR吗？

[ ] 是的，我愿意提交 PR！

行为守则

[ ] 我同意遵守该项目的行为准则

在第一次运行之前，必须先执行，如下： tar -zcvf /service/software/seatunnel/plugins.tar.gz /service/software/seatunnel/plugins and chmod 777 /service/software/seatunnel/plugins.tar.gz

This can solve the problem, chmod 777 should be written in the official document here

Nov 21 '22 08:11 wangsvip

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

Dec 22 '22 00:12 github-actions[bot]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

Dec 29 '22 00:12 github-actions[bot]

seatunnel seatunnel copied to clipboard

[Bug] [Spark] java.nio.file.AccessDeniedException: /service/software/seatunnel/plugins.tar.gz

Search before asking

What happened

SeaTunnel Version

SeaTunnel Config

Running Command

Error Exception

Flink or Spark Version

Java or Scala Version

Screenshots

Are you willing to submit PR?

Code of Conduct

Search before asking

What happened

SeaTunnel Version

SeaTunnel Config

Running Command

Error Exception

Flink or Spark Version

Java or Scala Version

Screenshots

Are you willing to submit PR?

Code of Conduct

询问前搜索

发生了什么

海隧道版本

海隧道配置

运行命令

错误异常

Flink 或 Spark 版本

Java 或 Scala 版本

截图

你愿意提交PR吗？

行为守则

seatunnel
seatunnel copied to clipboard