airbyte
airbyte copied to clipboard
[Low-Code CDK] Fix SubstreamSlicer parameter assignment and throw errors when RequestOption is path.
What is the problem
Originally identified from work on another PR here: https://github.com/airbytehq/airbyte/pull/20861#discussion_r1062971659
From the substream slicer, we currently allow for developers to attempt to inject values into the request path. This is quite misleading because under the hood we don't actually do anything with the path because stream slicers don't have access to the path. They are built onto the retriever whereas path is managed by requesters. We should just throw an error if they attempt to do that. We have no existing connectors that try to do this
We are also incorrectly assigning the stream_state_value for the slices being created by the substream slicer.
todo add an example.
What is the desired solution
Throw errors when path is configured. We should be properly assigning to the correct key in the stream slice based on the parent stream config's request option (not the stream_state_value which is meant to indicate where to pull state data from a record)
Acceptance Criteria
- An error should be thrown when a parent_stream_config's
request_option
,inject_into
is set to path - During
stream_slices()
, the resulting slice should store thestream_state_value
under a key based on theparent_stream_config.request_option.field_name