Mojmir Vinkler
Mojmir Vinkler
Refactor upserts to MySQL from `grapher://` step. This allows us to compare checksums of data & metadata by indicator and skip upserts to `variables` table in MySQL if metadata doesn't...
Refactor Step paths like channel / namespace / version / name. Move the logic from bespoke functions to steps themselves and use `Step` properties to access them. Add `CatalogPath` to...
We have a single case where public dataset (`data://garden/covid/latest/combined` and hence our full covid dataset) depends on private dataset `data-private://garden/covid/latest/sequence`. ``` data://garden/covid/latest/combined: - data://garden/covid/latest/testing - data://garden/covid/latest/cases_deaths - data-private://garden/covid/latest/sequence - data://garden/demography/2024-07-15/population...
## Problem Some grapher steps can take annoyingly long, especially when on poor internet connection. For instance ``` etlr grapher://grapher/minerals/2024-07-15/minerals --grapher --force --only ``` can take anywhere between 20s and...
## Problem We use the Jinja2 templating engine in our metadata YAML files, especially in cases with dimensions, to avoid repeating the same phrases over and over. The problem is...
Most of our grapher steps are just propagating the same dataset to the grapher channel. It's not a big deal, but it's one more dataset to store and maintain. We...
COVID [explorer config file](https://github.com/owid/etl/blob/master/etl/steps/export/explorers/covid/latest/covid.config.yml) and helper functions look very helpful. We should put them in `explorer_helpers.py` to make them available to others. We should also add JSON schema and update...
Our deployment process is unnecessarily complex right now. We haven't changed it when migrating from Netlify to Cloudflare and there easy ways how to simplify it: - [ ] Our...