dlt
dlt copied to clipboard
allow resources with many pipes (ie. partitioned resources)
Background Each resource has a single execution pipe that yields items when extracted. We want to support resources with many pipes that can yield items against the same resource schema. This allows to create resources that extract several files or several partitions of the same table easily.
Currently if source tries to create two resources with the same name, we get an error. The intended behavior was to create a resource with additional pipes.
Requirements
-
- [ ] allow to add secondary pipes to a resource
-
- [ ] update the source that if a resource with the same name is returned/added we add the pipe to the existing resource and apply hints from new resource (we can add table_variant if the new resource affects different table_name)
-
- [ ] update
DltResourceDict
to handle additional pipes. there's plenty of code that is accessing_pipe
property of a resource
- [ ] update
Tests We need to test several low level operations on the resources.
- cloning and calling
- iterating (as part of resources and source)
- async generators and parallel extraction should work (pipes should be parallelized)
-
DltResourceDict
tests must be extended to resources with multiple pipes. adding new resources, generating dag etc. are all to be checked - standard end to end tests