airbyte
airbyte copied to clipboard
[ISSUE #19961] first iteration of refactor stream slices
What
First steps in addressing #19961 . In this PR:
- Add
incrementalfor DatetimeBasedCursor at the retriever level - Change
stream_slicerto be an array
Notes:
- source-braze uses a custom component that is quite close to the concept of incremental and therefore, we allow custom implementation of incremental. However, the implementation is a little awkward since to implement an incremental, we need to implement
StreamSlicerright now - Some sources were defining DatetimeStreamSlicer but were not using them. See:
- source-coin-api
- source-punk-api
- source-twitter
- source-waiteraid
Concerns/questions:
- Is it ok if we still have the "stream_slice" concept exposed (like
stream_slice['start_time']) as part of theincremental? - Currently, we do not enforce that DatetimeStreamSlicer can't be inherited as a stream slicer. Is it ok?
Still TODO:
- Update the docs
- Move
incrementalon the stream level and renameincremental_sync- Add interface for incremental to always expose
cursor_field
- Add interface for incremental to always expose
- Rename
stream_slicerconcept toiterable- Update
declarative_component_schema - Move/rename streamslice package
- Update