seatunnel

seatunnel copied to clipboard

Search before asking

• [X] I had searched in the issues and found no similar issues.

What happened

启动seatunnel引擎之后，提交任务的出错 ，请问，localfile方式的部署，为啥还需要设置HADOOP_HOME and hadoop.home.dir呢？

SeaTunnel Version

2.3.4 windows环境

SeaTunnel Config

seatunnel:
  engine:
    history-job-expire-minutes: 1440
    backup-count: 1
    queue-type: blockingqueue
    print-execution-info-interval: 60
    print-job-metrics-info-interval: 60
    slot-service:
      dynamic-slot: true
    checkpoint:
      interval: 10000
      timeout: 60000
      storage:
        type: hdfs
        max-retained: 3
        plugin-config:
          storage.type: hdfs
          fs.defaultFS: file:/// # Ensure that the directory has written permission

Running Command

./bin/seatunnel-cluster.cmd -d

.\bin\seatunnel.cmd --config config\pg-es-test.conf

Error Exception

Caused by: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
        at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:549) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4]
        at org.apache.hadoop.util.Shell.getHadoopHomeDir(Shell.java:570) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4]
        at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:593) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4]
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:690) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4]
        at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:78) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4]
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3487) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4]
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3482) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4]
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4]
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4]
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4]
        at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.initStorage(HdfsStorage.java:68) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.<init>(HdfsStorage.java:57) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.checkpoint.storage.hdfs.common.HdfsFileStorageInstance.getOrCreateStorage(HdfsFileStorageInstance.java:53) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorageFactory.create(HdfsStorageFactory.java:75) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.<init>(CheckpointManager.java:104) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.server.master.JobMaster.initCheckPointManager(JobMaster.java:247) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.server.master.JobMaster.init(JobMaster.java:230) ~[seatunnel-starter.jar:2.3.4]
        at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$3(CoordinatorService.java:419) ~[seatunnel-starter.jar:2.3.4]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_391]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_391]
        ... 3 more

Zeta or Flink or Spark Version

Zeta 2.3.4

Java or Scala Version

java version "1.8.0_391"

Screenshots

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

https://seatunnel.incubator.apache.org/zh-CN/docs/2.3.1/seatunnel-engine/checkpoint-storage/

localfile的配置文档有问题吧，改成以下配置可以运行成功

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

官方文档还没有更新这个，可以麻烦更新下文档吗，https://seatunnel.apache.org/docs/2.3.5/seatunnel-engine/checkpoint-storage/

请问解决了吗？我遇到了同样问题

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.