HDDS-7328. Improve Deletion of FSO Paths

[Xushaohong]

What changes were proposed in this pull request?

Currently, the deleting speed of paths in the FSO bucket is too slow.

Every time the DirectoryDeletingService executes(every 1m by default),  it chooses only one path from DeletedDirectroyTable and tries adding its sub-files and sub-dirs into the PurgeDirectories Request. The real deletion happens in OMKeyDeleteResponseWithFSO, which moved all sub-dirs into DeletedDirectroyTable.  Then these sub-dirs will be chosen by the future execution of DirectoryDeletingService.

In the real production environment, such deletion speed is not applicable. There could be lots of dirs deleted without sub-dirs or sub-files. The dir deletion speed would lag behind the dir creation speed.

In this PR, I propose to optimize the logic of the DirectoryDeletingService, we could consume all the quota (ozone.path.deleting.limit.per.task) to delete paths as much as possible. The good news is the PB is already designed to do so, we could add more PurgePathRequest in one PurgeDirectoriesRequest.

Originally: 1 min. At most 1 path from deletedDirectoryTable. Now: 1min. At most 10000 paths from deletedDirectoryTable.

message PurgeDirectoriesRequest {
  repeated PurgePathRequest deletedPath = 1;
}

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7328

How was this patch tested?

UT and real env

[Xushaohong]

@linyiqun @rakeshadr PTAL.

This should be compatible with HDDS-5048.

This aims to improve the efficiency of the deletion if not enough sub-files or sub-dirs are chosen.

[Xushaohong]

Thx @JacksonYao287 for the review!

[adoroszlai]

@JacksonYao287 please don't forget to close the Jira issue and set the fix version based on where it was committed.

[kaijchen]

Thanks @Xushaohong for the improvement and @JacksonYao287 for the review. I have backported this fix into 1.3 branch.