posthog icon indicating copy to clipboard operation
posthog copied to clipboard

Make all clickhouse migrations async migrations

Open macobo opened this issue 2 years ago • 1 comments

Is your feature request related to a problem?

We now have django migrations, infi.orm clickhouse migrations and async migrations. Of these, clickhouse migrations and async migrations have complex inter-dependencies. This causes many problems:

  1. infi.orm kind of sucks - it doesn't report bugs properly, users run into issues with mysterious errors like Table default.infi_clickhouse_orm_migrations doesn't exist
  2. Inter-depencies can cause bugs: we had 0004 replication migration as non-mandatory for a while, but a later infi.orm migration implicitly depended on it being complete, causing issues for deployments which did not have it as complete.
  3. Related to previous, it's hard for migration authors outside of team ingestion to know what they can rely on in clickhouse migrations.

Describe the solution you'd like

  • Convert all infi.orm migrations to async migrations, with correct dependencies between various migrations
  • Make it possible to autorun certain migrations automatically during bin/migrate
  • Improve UX of async migrations page to handle 30+ migrations (e.g. hiding completed migrations by default)

Additional context

cc @guidoiaquinti @hazzadous who have run into issues with the Table missing error cc @tiina303 and @yakkomajuri from team ingesiton

Thank you for your feature request – we love each and every one!

macobo avatar Aug 04 '22 07:08 macobo

infi.orm kind of sucks - it doesn't report bugs properly, users run into issues with mysterious errors like Table default.infi_clickhouse_orm_migrations doesn't exist

I'm hitting this problem right now with a fresh install so if you have more details I'm interested :)

allan-simon avatar Sep 15 '22 20:09 allan-simon