[Bug] [Source & Sink] After the local file is read and sinked, there is duplicate data

[CCweixiao]

Search before asking

• [X] I had searched in the issues and found no similar issues.

What happened

export version="2.3.5" wget "https://archive.apache.org/dist/seatunnel/${version}/apache-seatunnel-${version}-bin.tar.gz" tar -zxvf apache-seatunnel-2.3.5-bin.tar.gz cd apache-seatunnel-2.3.5 sh bin/install-plugin.sh 2.3.5 cd config cp v2.batch.config.template local.file.config.template

config content is: env { parallelism = 10 job.mode = "BATCH" }

source { LocalFile { path = "/tmp/input" file_format_type = "text" field_delimiter = "," schema = { fields { name = "string" age = "int", address = "string" } } } }

sink { LocalFile { path = "/tmp/output" file_format_type = "text" field_delimiter = "," } }

input source files are:

exec commond ./bin/seatunnel.sh --config ./config/local.file.config.template -e local

output sink files are:

SeaTunnel Version

2.3.5

SeaTunnel Config

env {
  parallelism = 10
  job.mode = "BATCH"
}

source {
  LocalFile {
    path = "/tmp/input"
    file_format_type = "text"
    field_delimiter = ","
    schema = {
      fields {
        name = "string"
        age = "int",
        address = "string"
      }
    }
  }
}

sink {
  LocalFile {
    path = "/tmp/output"
    file_format_type = "text"
    field_delimiter = ","
  }
}

Running Command

./bin/seatunnel.sh --config ./config/local.file.config.template -e local

Error Exception

Duplicate data

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions[bot]]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.