owid-grapher icon indicating copy to clipboard operation
owid-grapher copied to clipboard

Published 20 hours ago • verified publisher icon owid

Migrations that remove columns break ETL

Open Marigold opened this issue 10 months ago • 3 comments

ETL uses its own ORM. The problem is that if someone removes a column with a migration in grapher, ETL breaks. This happened recently with isBulkImport removal that broke fasttrack (cc @marcelgerber).

Any ideas on how to make it more resilient? (I don't have any yet)

Marigold avatar Apr 25 '24 14:04 Marigold

Hey Mojmir! I was wondering if we could have a bot without too much trouble that checks prs in the owid-grapher repo and whenever there is a push checks if there were changes in ./db/migrations/. If so, then it adds this checklist as markdown to the PR description so that you are reminded to check all points and see if they apply.

What do you think?

danyx23 avatar Apr 28 '24 13:04 danyx23

Posting a checklist to PR would be super neat! I plan to work on this soon, I'll make sure to add it.

Marigold avatar Apr 30 '24 09:04 Marigold

Hey Mojmir! I was wondering if we could have a bot without too much trouble that checks prs in the owid-grapher repo and whenever there is a push checks if there were changes in ./db/migrations/.

Just noting that this part of the proposal is super easy; see https://github.com/owid/owid-grapher/blob/dff12bf7e17aaf441e8c2a00a089df7c31c2c7dc/.github/workflows/sync-grapher-schema-to-digital-ocean.yml#L6-L7 for example.

marcelgerber avatar May 01 '24 11:05 marcelgerber