hdfs icon indicating copy to clipboard operation
hdfs copied to clipboard

Allow overwrites during rename. This is required for doing atomic puts.

Open Apteryx0 opened this issue 1 year ago • 0 comments

To implement atomic writes/puts, you need to write to a temporary file (non-atomic operation) and then rename the temporary file to the destination in a single (atomic) operation. Since rename does not currently support this (i.e. if the destination file already exists rename will fail and thus a delete and then rename is required, which is not atomic), added an option to pass the renameoptions="overwrite" to the WebHDFS rename API. This is received by this code here: https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L749

Which in turn calls: https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java#L545

Which is documented as being atomic.

Haven't updated or run any hdfs tests - this is tested and works though in my own code base.

Apteryx0 avatar Jun 12 '24 15:06 Apteryx0