posthog
posthog copied to clipboard
Make all clickhouse migrations async migrations
Is your feature request related to a problem?
We now have django migrations, infi.orm clickhouse migrations and async migrations. Of these, clickhouse migrations and async migrations have complex inter-dependencies. This causes many problems:
- infi.orm kind of sucks - it doesn't report bugs properly, users run into issues with mysterious errors like
Table default.infi_clickhouse_orm_migrations doesn't exist
- Inter-depencies can cause bugs: we had 0004 replication migration as non-mandatory for a while, but a later infi.orm migration implicitly depended on it being complete, causing issues for deployments which did not have it as complete.
- Related to previous, it's hard for migration authors outside of team ingestion to know what they can rely on in clickhouse migrations.
Describe the solution you'd like
- Convert all infi.orm migrations to async migrations, with correct dependencies between various migrations
- Make it possible to autorun certain migrations automatically during bin/migrate
- Improve UX of async migrations page to handle 30+ migrations (e.g. hiding completed migrations by default)
Additional context
cc @guidoiaquinti @hazzadous who have run into issues with the Table missing error cc @tiina303 and @yakkomajuri from team ingesiton
Thank you for your feature request – we love each and every one!
infi.orm kind of sucks - it doesn't report bugs properly, users run into issues with mysterious errors like Table default.infi_clickhouse_orm_migrations doesn't exist
I'm hitting this problem right now with a fresh install so if you have more details I'm interested :)