etl
etl copied to clipboard
🎉 Add new explorer:// step for creating TSV files
Follow up for https://github.com/owid/etl/pull/2974
Add new ExplorerStep
named explorer://channel/namespace/version/short_name
. The step calls create_explorer
function, which creates a TSV file in owid-content
repository. Check our discussion in Notion.
Implementation:
- The step is under
etl/steps/explorer
instead ofetl/steps/data
(it was necessary to makePathFinder
work) - Explorer step runs every time (i.e. its
is_dirty
method returns True). We could change this in the future by creating metadata JSON file with checksum and comparing it withchecksum_input
(similarly how we do it forDataStep
) - Reuse methods of
DataStep
- this isn't the cleanest way, but could be refactored in the future and prevents duplicating code
TODO:
- [ ] Make it work against staging servers (explorer step would update TSV file on the server, while still being on the master branch)