etl 📊 Update electricity and energy data

📊 Update electricity and energy data

Open pabloarosado opened this issue 9 months ago • 1 comments

Update Ember's global electricity data, and update all energy datasets.

NOTE: This data is embargoed until the 8th of May at midnight UK time.

Fixes: https://github.com/owid/owid-issues/issues/1401

Main changes

Previously, we were fetching European data (from Ember's European Electricity Review) and global data (from Ember's Yearly Electricity Data) separately, and combining them in a garden step. Now, Ember is publishing all this data together as part of the Yearly Electricity Data, which greatly simplifies our pipeline.

Apart from this, I also did minor adaptations to the old steps (like using the latest income groups data).

I also bumped dependencies of energy steps (where the data has not changed) and archived unused energy steps.

TO-DO (before the 8th of May):

[x] Fix issue with lower-middle-income countries (see this chart).
[x] Archive unused steps and update other step dependencies.
[x] Use the latest Maddison GDP data, if published before this.
[ ] Implement changes suggested in this PR review, if any.

TO-DO (on the 8th of May):

[ ] Change private steps to public steps in the dag.
[ ] Adapt yearly_electricity snapshot to fetch data from a URL instead of a local file.
[ ] Re-run snapshot in public mode, and push changes to this branch.
[ ] Merge this branch to master.
[ ] Use chart-sync tool to update charts in production.
[ ] Update data in the energy-data repos (and upload latest data files to S3).
[ ] Update README in co2-data repos (no need to update data, since it hasn't changed).