sdk icon indicating copy to clipboard operation
sdk copied to clipboard

Follow-up from "Resolve "Bug: `get_starting_timestamp()` and `get_starting_replication_key_value()` should be cached""

Open MeltyBot opened this issue 4 years ago • 0 comments

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/232

Originally created by @aaronsteers on 2021-10-07 19:01:48


The following discussion from !157 should be addressed:

  • [ ] @aaronsteers started a discussion: (+2 comments)

    @edgarrmondragon - This is looking great. I just did another review and there's just one remaining change I'd like to request. Can you add explicit removal of the "starting" value's key during the "finalize" step here: https://gitlab.com/meltano/sdk/-/blob/1378ed3ede701d2edde1d96fbd1db09d2971e3c4/singer_sdk/helpers/_state.py#L242

    Since this is one of those tokens that's only relevant while the stream is in-flight, we should explicitly remove it at the end of the sync operation. (Which is the function of finalize_state_progress_markers().)

    For unsorted streams, I think it may automatically show up under "progress_markers" which gets purged as a whole object during finalization. However, for "normal" sorted streams, we won't have a "progress_markers" dict and we should remove the start value if present.

MeltyBot avatar Oct 07 '21 19:10 MeltyBot