airbyte icon indicating copy to clipboard operation
airbyte copied to clipboard

[Low-Code CDK] Fix SubstreamSlicer parameter assignment and throw errors when RequestOption is path.

Open brianjlai opened this issue 2 years ago • 0 comments

What is the problem

Originally identified from work on another PR here: https://github.com/airbytehq/airbyte/pull/20861#discussion_r1062971659

From the substream slicer, we currently allow for developers to attempt to inject values into the request path. This is quite misleading because under the hood we don't actually do anything with the path because stream slicers don't have access to the path. They are built onto the retriever whereas path is managed by requesters. We should just throw an error if they attempt to do that. We have no existing connectors that try to do this

We are also incorrectly assigning the stream_state_value for the slices being created by the substream slicer.

todo add an example.

What is the desired solution

Throw errors when path is configured. We should be properly assigning to the correct key in the stream slice based on the parent stream config's request option (not the stream_state_value which is meant to indicate where to pull state data from a record)

Acceptance Criteria

  • An error should be thrown when a parent_stream_config's request_option, inject_into is set to path
  • During stream_slices(), the resulting slice should store the stream_state_value under a key based on the parent_stream_config.request_option.field_name

brianjlai avatar Jan 06 '23 05:01 brianjlai