hadoop-connectors icon indicating copy to clipboard operation
hadoop-connectors copied to clipboard

Hadoop FileSystem delete method returns false even though file is actually deleted

Open selimelawwa opened this issue 2 years ago • 0 comments

In my Java application I have an implementation for a file-system, where my file class is a wrapper for Hadoop filesystem methods. I am upgrading the from hadoop3-1.9.17 to hadoop3-2.2.8 and I am using the shaded jar of the new version.

My delete method implementation is as follows:

@Override
public boolean delete(boolean recursively) {
    return fs.delete(path, recursively);
}

Where fs is an instance of org.apache.hadoop.fs.FileSystem so this method calls org.apache.hadoop.fs.FileSystem.delete(Path, boolean)

Frequently when calling the above delete method on a directory which contains files it returns false 'meaning dir/file NOT deleted' however when checking the directory is actually deleted at Google Cloud Storage.

Is there a change in the upgraded GCS connector causing this (caching of file-status i.e) or is there a configuration to fix this issue? I also created this [stackoverflow question](https://stackoverflow.com/questions/74445091/gcs-connector-hadoop3-hadoop3-2-2-8-filesystem-delete-returns-false-but-file-dir but got no answers

selimelawwa avatar Nov 22 '22 12:11 selimelawwa