etl

etl copied to clipboard

:tada: engineering: add prefect engine when running ETL

13

Adds option `--engine` for specifying which scheduler to use (default is `--engine etl`). Using `--engine prefect` orchestrates ETL with Prefect and creates SQLite file that could be inspected with Prefect...

Marigold

:sparkles: Less verbose logs & fix warnings

1

Marigold

📊 Update World Bank income groups

1

Main issue https://github.com/owid/owid-issues/issues/1625 When updating income groups, I am not sure if I should replace in every step that is using them? ~I thought it was the right thing to...

paarriagadap

:construction: Draft PR for branch temp-world-bank-groups-2024

1

paarriagadap

📊 ai: Epoch maximum compute, parameters and data test

1

veronikasamborska1994

We're updating monkeypox data from [owid-admin-prod](https://github.com/owid/ops/blob/main/templates/owid-admin-prod/crontab#L35) and schedule it with CRON. It's hidden and shouldn't run there, so the best would be migrating it to ETL.

Marigold

Nightly build fails on multiple steps

1

We have failures over multiple days on the nightly build: https://buildkite.com/our-world-in-data/etl-full-private-rebuild-nightly Several steps appear to be failing. ### `garden/harvard/2023-09-18/colonial_dates_dataset` Traceback ``` bk;t=1722213749841^GTraceback (most recent call last): bk;t=1722213749841^G File "", line...

larsyencken

🚧 staging server: Tuna

1

antea04

🎉 Add new explorer:// step for creating TSV files

1

Follow up for https://github.com/owid/etl/pull/2974 Add new `ExplorerStep` named `explorer://channel/namespace/version/short_name`. The step calls `create_explorer` function, which creates a TSV file in `owid-content` repository. Check our [discussion in Notion](https://www.notion.so/owid/2024-Data-architecture-60c39d558abf4eb2973cc52447d066fb?pvs=4#2eda2f8fc43b4cefb2ee6aa3f82a128e). ## Implementation: -...

Marigold

📊 ai: archive old datasets

1

veronikasamborska1994