etl icon indicating copy to clipboard operation
etl copied to clipboard

Version Tracker complains because `grapher://grapher` steps are not found in the dag

Open pabloarosado opened this issue 5 months ago • 0 comments

Problem

Currently, running etl d version-tracker raises an error, because, e.g.

* Missing step
    grapher://grapher/energy/2024-06-20/primary_energy_consumption
  is a dependency of the following active steps:
    export://multidim/energy/latest/energy

Expected behaviour

We would expect that version tracker automatically includes the auto-generated grapher://grapher/... steps when sanity checking the dependency graph, i.e. no error here.

Why this is happening

This happens because the grapher://grapher/ dependency is not in the dag (however, the corresponding data://grapher/ step is in the dag, so the error should not be raised). This started happening recently, since we started having export steps that depend on grapher://grapher steps.

Technical notes

  • We note that we really do need to execute grapher://grapher/... steps before exporting
  • This means version tracker needs to build off a computed DAG that includes these steps

pabloarosado avatar Sep 10 '24 15:09 pabloarosado