OpenSearch icon indicating copy to clipboard operation
OpenSearch copied to clipboard

Add support to skip pinned timestamp in remote translog garbage collector

Open sachinpkale opened this issue 5 months ago • 5 comments

Description

  • As part of shallow snapshot optimisations, we are introducing timestamp based implicit locking support. This feature will be called Timestamp Pinning in remote backed storage. When a timestamp is pinned, remote store garbage collectors will skip deleting data corresponding to the timestamp.
  • Changes to segment store garbage collection are made as part of PR: https://github.com/opensearch-project/OpenSearch/pull/15017
  • In this PR, we are making following changes to translog garbage collection:
    • Fetch metadata files in reverse-chronological order
    • Filter out metadata files that match with pinned timestamp and metadata files that are created post successful fetch of pinned timestamp.
    • From the remaining metadata files, read min and max generation and filter out generations by pinned timestamp metadata files.

Related Issues

  • https://github.com/opensearch-project/OpenSearch/issues/15064

Check List

  • [x] Functionality includes testing.
  • [ ] API changes companion pull request created, if applicable.
  • [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

sachinpkale avatar Aug 26 '24 10:08 sachinpkale