debussy_concert

debussy_concert copied to clipboard

Possibility to do a partial reprocessing of a DAG

1

The problem: Currently, our examples are heavily dependent on `dag_run.start_date` as the metadata used for `_ingestion_ts` And this add an issue that we can't have a partial reprocessing as the...

NiltonDuarte

Metadata long term solution

2

Draft PR for the long term solution of the metadata schema issue This PR aims to add a is_metadata field to the generic table schema, this PR also decouple the...

NiltonDuarte

Long term solution for BigQuery load with schema

1

PR https://github.com/DotzInc/debussy_concert/pull/28 addresses a quick solution As a long-term solution we should add a `is_metadata: bool (default to False)` attribute to the schema file. I will add more info to...

NiltonDuarte

Parameter extract_query_from_temp from Reverse ETL composition does not support setting a .sql file. This query need to get the value of reverse_etl_table_uri that is not passed/parsed by jinja and is...

NiltonDuarteAtDotz