airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

[airbyte-cdk] Fix a bug so that successful Python RFR streams are not synced on subsequent attempts

Open brianjlai opened this issue 9 months ago • 1 comments

What

When I first implemented RFR for Python, I was using the empty state value {} to indicate the sync was complete and should end. However, we need to emit a terminal sentinel value. That way if the sync fails on another stream, on the subsequent attempt, we don't try to resync the previously successful streams.

How

By emitting the terminal state message for an RFR stream {"__ab_full_refresh_sync_complete": True}, we can opt to skip attempting the stream read effectively moving onto the next stream. We still emit the proper stream status messages back to the platform.

User Impact

None. RFR is not supported in the platform yet and it will always drop full refresh state before an attempt.

Can this PR be safely reverted and rolled back?

  • [x] YES 💚
  • [ ] NO ❌

brianjlai avatar May 23 '24 10:05 brianjlai