spark-distcp icon indicating copy to clipboard operation
spark-distcp copied to clipboard

Published 20 hours ago • verified publisher icon CoxAutomotiveDataSolutions

Handle missing directories

Open gquintana opened this issue 3 years ago • 2 comments

Trying to fix #10.

I introduced a configurable "missing directory" strategy:

  • "fail" is default behaviour (HDFS)
  • "create" auto creates target directory if not exist (HDFS)
  • "log" doesn't fail nor create directories because it's not needed (S3, Ceph and other object storage)

Apologies for code quality, I am not fluent in Scala, tell me how to fix and improve.

gquintana avatar Feb 05 '22 11:02 gquintana

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 38 Code Smells

No Coverage information No Coverage information
33.5% 33.5% Duplication

sonarqubecloud[bot] avatar Feb 08 '22 11:02 sonarqubecloud[bot]

I did my best to reduce code duplication and comply with SonarCloud check. Sadly I could achieve it. Note that I did'nt introduce most of this duplicated code.

gquintana avatar Feb 08 '22 11:02 gquintana