dlt icon indicating copy to clipboard operation
dlt copied to clipboard

Published 20 hours ago • verified publisher icon dlt-hub

[experiment] Polars-based JSON normalizer

Open jorritsandbrink opened this issue 1 year ago • 1 comments

Hacked-together rudimentary Polars-based JSON normalizer as experiment to learn what it takes to replace current normalizer and which speedups to expect.

  • approach: process single JSON line with original normalizer to infer/create dlt schema, then process all JSON lines with Polars to create normalized files
  • basic functionality seems to work, but there's no feature parity with current normalizer
  • Polars code can probably be optimized to run faster

image

jorritsandbrink avatar Sep 09 '24 15:09 jorritsandbrink

Deploy Preview for dlt-hub-docs ready!

Name Link
Latest commit 79c03ac62cbe31599cea3dcc37abaa371618eabd
Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/66df0f8b79097e0008c3d16b
Deploy Preview https://deploy-preview-1793--dlt-hub-docs.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify[bot] avatar Sep 09 '24 15:09 netlify[bot]