etl icon indicating copy to clipboard operation
etl copied to clipboard

Remove redundant grapher steps

Open Marigold opened this issue 5 months ago • 1 comments

Most of our grapher steps are just propagating the same dataset to the grapher channel. It's not a big deal, but it's one more dataset to store and maintain. We had some ideas for simplification:

  1. Instead of
grapher://grapher/namespace/version/dataset:
  - data://grapher/namespace/version/dataset
  
data://grapher/namespace/version/dataset
  - data://garden/namespace/version/dataset

simply build grapher from garden dataset

grapher://grapher/namespace/version/dataset:
  - data://garden/namespace/version/dataset
  1. If there's a YAML file in step directory without *.py file, take the garden dataset and enhance it with the YAML file

Marigold avatar Aug 26 '24 16:08 Marigold