etl icon indicating copy to clipboard operation
etl copied to clipboard

🎉 Add new explorer:// step for creating TSV files

Open Marigold opened this issue 7 months ago • 1 comments

Follow up for https://github.com/owid/etl/pull/2974

Add new ExplorerStep named explorer://channel/namespace/version/short_name. The step calls create_explorer function, which creates a TSV file in owid-content repository. Check our discussion in Notion.

Implementation:

  • The step is under etl/steps/explorer instead of etl/steps/data (it was necessary to make PathFinder work)
  • Explorer step runs every time (i.e. its is_dirty method returns True). We could change this in the future by creating metadata JSON file with checksum and comparing it with checksum_input (similarly how we do it for DataStep)
  • Reuse methods of DataStep - this isn't the cleanest way, but could be refactored in the future and prevents duplicating code

TODO:

  • [ ] Make it work against staging servers (explorer step would update TSV file on the server, while still being on the master branch)

Marigold avatar Jul 26 '24 11:07 Marigold