dlt
dlt copied to clipboard
rest_api: use @defer decorator on transformer loop for parallel execution
Source name
rest_api
Describe the data you'd like to see
Dependent resources loop over all data items in a batch sent from the connected parent. We can easily parallelize this loop.
- detect if
parallelize
flag is enabled on a transformer when generating loop over items - if so use decorated version of the function (via dlt.defer) in essence such decorator wraps transformer function in callables and many of those can be executed at the same time
Are you a dlt user?
Yes, I'm already a dlt user.
Do you ready to contribute this extension?
Yes, I'm ready.
dlt destination
No response
Additional information
No response