OpenSearch
OpenSearch copied to clipboard
Add support to skip pinned timestamp in remote translog garbage collector
Description
- As part of shallow snapshot optimisations, we are introducing timestamp based implicit locking support. This feature will be called Timestamp Pinning in remote backed storage. When a timestamp is pinned, remote store garbage collectors will skip deleting data corresponding to the timestamp.
- Changes to segment store garbage collection are made as part of PR: https://github.com/opensearch-project/OpenSearch/pull/15017
- In this PR, we are making following changes to translog garbage collection:
- Fetch metadata files in reverse-chronological order
- Filter out metadata files that match with pinned timestamp and metadata files that are created post successful fetch of pinned timestamp.
- From the remaining metadata files, read min and max generation and filter out generations by pinned timestamp metadata files.
Related Issues
- https://github.com/opensearch-project/OpenSearch/issues/15064
Check List
- [x] Functionality includes testing.
- [ ] API changes companion pull request created, if applicable.
- [ ] Public documentation issue/PR created, if applicable.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.