spark-distcp icon indicating copy to clipboard operation
spark-distcp copied to clipboard

feature: Implement -direct option for S3 and other object stores

Open gquintana opened this issue 2 years ago • 0 comments

https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html#Command_Line_Options

-direct | Write directly to destination paths | Useful for avoiding potentially very expensive temporary file rename operations when the destination is an object store

https://github.com/CoxAutomotiveDataSolutions/spark-distcp/blob/v0.2.5/src/main/scala/com/coxautodata/utils/CopyUtils.scala#L380

gquintana avatar Mar 18 '22 10:03 gquintana