etl
etl copied to clipboard
Version Tracker complains because `grapher://grapher` steps are not found in the dag
Problem
Currently, running etl d version-tracker
raises an error, because, e.g.
* Missing step
grapher://grapher/energy/2024-06-20/primary_energy_consumption
is a dependency of the following active steps:
export://multidim/energy/latest/energy
Expected behaviour
We would expect that version tracker automatically includes the auto-generated grapher://grapher/...
steps when sanity checking the dependency graph, i.e. no error here.
Why this is happening
This happens because the grapher://grapher/
dependency is not in the dag (however, the corresponding data://grapher/
step is in the dag, so the error should not be raised). This started happening recently, since we started having export
steps that depend on grapher://grapher
steps.
Technical notes
- We note that we really do need to execute
grapher://grapher/...
steps before exporting - This means version tracker needs to build off a computed DAG that includes these steps