data-prepper icon indicating copy to clipboard operation
data-prepper copied to clipboard

Published 20 hours ago • verified publisher icon opensearch-project

Track last scan time from before scan starts instead of based on last Modified of objects

Open graytaylor0 opened this issue 9 months ago • 0 comments

Description

This change makes it so that the filtering done between scans for last modified objects, which is used to reduce lookups to the coordination store, is tracked based on the time before the listObjects call to scan for each bucket is made. This will make it impossible to miss items that are uploaded in the middle or end of a scan

Issues Resolved

Previously attempted to fix with https://github.com/opensearch-project/data-prepper/pull/4124

Original issue: https://github.com/opensearch-project/data-prepper/issues/4123

Check List

  • [x] New functionality includes testing.
  • [ ] New functionality has a documentation issue. Please link to it in this PR.
    • [ ] New functionality has javadoc added
  • [x] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

graytaylor0 avatar May 02 '24 04:05 graytaylor0