ozone
ozone copied to clipboard
apache
HDDS-7258. Cleanup the allocated but uncommitted blocks
What changes were proposed in this pull request?
We found that the blocks the client committed may not be equal to the allocated(when EC stripe fails), and the uncommitted blocks would remain forever as orphan blocks as no key maps to them anymore.
Here I use a tricky solution that considers these blocks as repeated/ overwritten key blocks, and thus they would be deleted in the existing deletion path.
https://docs.google.com/document/d/1Oi1zPKdmvA7DFwIcUWz6zw_p-pY3D1L76gyV37jQT94/edit#
The prototype of the design.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7258
How was this patch tested?
UT and Manual test in the test environment.
Could you please describe how these blocks are being detected in the PR description - like a small design overview?
Could you please describe how these blocks are being detected in the PR description - like a small design overview?
Sure, will update later, I am working on fixing some issues found by UT.
https://docs.google.com/document/d/1Oi1zPKdmvA7DFwIcUWz6zw_p-pY3D1L76gyV37jQT94/edit#
The prototype of the design.
Thanks for the work @Xushaohong. Is it possible to delete those blocks directly instead of relying on OpenKeyCleanupService? I'm worrying about too much coupling here.
Thx for the question~ @kaijchen The case for OpenKeyCleanupService is not related to the logic here. They should be two cases as I mentioned in the DOC.
Thx for the question~ @kaijchen The case for OpenKeyCleanupService is not related to the logic here. They should be two cases as I mentioned in the DOC.
Sorry I misread your purposal. The pseudoKeyInfo is moved into DeleteKeyTable instead of staying at OpenKeyTable, which sounds reasonable to me.
Thanks @Xushaohong for updating the patch. Filtering uncommitted blocks in verifyAndGetKeyLocations is a good idea. For clearity, it's better to return a Pair<updatedBlockLocations, uncommittedBlocks> instead, or create a inner class.