etl
etl copied to clipboard
Skip chart revision for charts with identical data
What
Improve the chart revision flow by skipping charts that have no differences.
Why? Why now?
Currently, charts are queued up for manual review even when there are no changes, e.g. because other variables in a dataset changed.
We should make this flow more efficient by only requiring QA when there is something to QA.
Technical notes
- Aren't there already options for auto-applying some chart revisions in staging sync?
- Yes, in staging sync this is to avoid a double approval workflow
- The chart revision code is the last piece for Daniel to migrate TypeORM-to-Knex database code; should he do it, or should we migrate it to Streamlit? Unclear
- Decision: we should do this not in the chart approval code but in the code that decides whether to queue a chart for approval
~~⏸️ We decided to park this until the next data engineering cycle, when we might migrate chart revision to Streamlit. We will also push for #data-viz
time to get Grapher redistributable, so that owid-grapher-py
can be our interface there.~~
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
This is simple to do and still useful
But maybe we should move it over to the grapher repo as this is where this logic lies?
We plan to deprecate chart revisions in favour of chart-diff (which already has this feature). What is "worth QA-ing" in general is a bigger question we shall open soon after.
Now it's a problem for chart-diff
, we can track it there.