dlt icon indicating copy to clipboard operation
dlt copied to clipboard

rest_api: use @defer decorator on transformer loop for parallel execution

Open rudolfix opened this issue 9 months ago • 0 comments

Source name

rest_api

Describe the data you'd like to see

Dependent resources loop over all data items in a batch sent from the connected parent. We can easily parallelize this loop.

  1. detect if parallelize flag is enabled on a transformer when generating loop over items
  2. if so use decorated version of the function (via dlt.defer) in essence such decorator wraps transformer function in callables and many of those can be executed at the same time

Are you a dlt user?

Yes, I'm already a dlt user.

Do you ready to contribute this extension?

Yes, I'm ready.

dlt destination

No response

Additional information

No response

rudolfix avatar May 21 '24 15:05 rudolfix