airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

Published 20 hours ago • verified publisher icon airbytehq

Improve SubstreamResumableFullRefreshCursor performance

Open szubster opened this issue 6 months ago • 3 comments

What

After https://github.com/airbytehq/airbyte/pull/42966 performance of github source dropped dramatically

How

Put partition information in values of _per_partition_state so it does not have to be deserialised every time.

Review guide

  1. substream_resumable_full_refresh_cursor.py

User Impact

Better performance

Can this PR be safely reverted and rolled back?

  • [x] YES 💚
  • [ ] NO ❌

szubster avatar Aug 19 '24 22:08 szubster