dlt icon indicating copy to clipboard operation
dlt copied to clipboard

feat(pipeline): implement normalize progress tracking for JSON normalizer

Open IlyaFaer opened this issue 1 year ago • 3 comments

Towards #1567

Implements item 1 for JsonLItemsNormalizer

IlyaFaer avatar Jul 26 '24 09:07 IlyaFaer

Deploy Preview for dlt-hub-docs ready!

Name Link
Latest commit 96b98830df1f67cc02fae601effd2aaf1825746c
Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/66c307272747fe000881e2f6
Deploy Preview https://deploy-preview-1639--dlt-hub-docs.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify[bot] avatar Jul 26 '24 09:07 netlify[bot]

For the case when we're first extracting data and then normalizing, it works fine

image

The code is chained, so it requires to through some args through several functions/classes. Tried to reduce it as much as possible.

Thinking about how to test it... Seems to me something cunning is needed. However, the code is working for the case when we're extracting and then normalizing data.

IlyaFaer avatar Jul 26 '24 09:07 IlyaFaer

collected metrics will not go through process boundary, we'll need different approach here

rudolfix avatar Sep 11 '24 09:09 rudolfix