spark-distcp
spark-distcp copied to clipboard
feature: Implement -direct option for S3 and other object stores
https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html#Command_Line_Options
-direct | Write directly to destination paths | Useful for avoiding potentially very expensive temporary file rename operations when the destination is an object store
https://github.com/CoxAutomotiveDataSolutions/spark-distcp/blob/v0.2.5/src/main/scala/com/coxautodata/utils/CopyUtils.scala#L380