airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

[ISSUE #19961] first iteration of refactor stream slices

Open maxi297 opened this issue 2 years ago • 0 comments

What

First steps in addressing #19961 . In this PR:

  • Add incremental for DatetimeBasedCursor at the retriever level
  • Change stream_slicer to be an array

Notes:

  • source-braze uses a custom component that is quite close to the concept of incremental and therefore, we allow custom implementation of incremental. However, the implementation is a little awkward since to implement an incremental, we need to implement StreamSlicer right now
  • Some sources were defining DatetimeStreamSlicer but were not using them. See:
    • source-coin-api
    • source-punk-api
    • source-twitter
    • source-waiteraid

Concerns/questions:

  • Is it ok if we still have the "stream_slice" concept exposed (like stream_slice['start_time']) as part of the incremental?
  • Currently, we do not enforce that DatetimeStreamSlicer can't be inherited as a stream slicer. Is it ok?

Still TODO:

  • Update the docs
  • Move incremental on the stream level and rename incremental_sync
    • Add interface for incremental to always expose cursor_field
  • Rename stream_slicer concept to iterable
    • Update declarative_component_schema
    • Move/rename streamslice package

maxi297 avatar Feb 01 '23 18:02 maxi297