rudolfix
rudolfix
**Background** Progress reporting in `normalize` and `load` steps are far from perfect. 1. in `normalize` we report progress on file level but that only is updated when a worker process...
**Background** Currently nested types (arrays, maps, records), if present in arrow schema, are mapped into `complex` `dlt` type. `complex` types are stored in JSON types in the destination (if available)...
### Documentation description `dlt` uses certain environment variables that are not a part of standard configuration. they are compiled in a separate module https://deploy-preview-1536--dlt-hub-docs.netlify.app/docs/api_reference/common/known_env we should link to it (maybe?)...
### Description We discovered peculiar problem with `rest_api` when users were passing a list of resources to the `run` function that contained a resource and a transformer: ```python page =...
## `0.5.1` release notes We have a few interesting changes coming in and we decided to increase `dlt` version to `0.5` which is a major release in our versioning scheme....
**Background** Building on #746 we want a robust fingerprinting and destination info to include in `LoadInfo` and pipeline traces. The fingerprinting is used to anonymously identify cloud destinations and allows...
**Background** We often deal with large amount of structured data that we want to load into destination. Ie. parquet or csv files (let's skip jsonl files for now). Currently `dlt`...
**Discussion** This actually brings back initial `dlt` behavior when it was used to process streams of data (but now we have state so the problem got harder). The idea is...
### dlt version 0.4.12 ### Describe the problem `dlt` creates a new working folder in a `/tmp` for each (1) dag creation (2) each actual dag execution. in case of...
**Background** **split pipeline** - create pipeline with a new name and move indicated sources/resources/tables to it. extracted and normalized files belonging to resources' tables are moved, schemas and state are...