chart icon indicating copy to clipboard operation
chart copied to clipboard

Run db:migrate in pre- and post-install/upgrade (#18, #26)

Open angdraug opened this issue 2 years ago • 10 comments

angdraug avatar Jan 15 '23 23:01 angdraug

Not sure if it shows up for you, but GitHub tells me:

First-time contributors need a maintainer to approve running workflows. Learn more.

I don't think this PR is going to land until someone does the needful.

angdraug avatar Jan 26 '23 20:01 angdraug

Please can you release this PR? It seems ok. Otherwise, this chart cannot be used within a highly automated environment with Terraform + Helm; the helm installation never ends, and the migration job is never triggered by Helm, making it impossible to use it for a new fresh install.

cc @dunn

paolomainardi avatar Jan 27 '23 18:01 paolomainardi

I'm not actually a maintainer of this repo, so I can't merge.

dunn avatar Jan 27 '23 21:01 dunn

I'm not actually a maintainer of this repo, so I can't merge.

Ops, so sorry, I saw your comments and just thought you was a maintainer too.

paolomainardi avatar Jan 27 '23 21:01 paolomainardi

Just tried this PR, and it doesn't work, the migrate job requires PVC already created otherwise the job cannot be executed.

The question is, does the migration job requires the PVC ?

paolomainardi avatar Jan 29 '23 15:01 paolomainardi

I tried again using a bucket instead of PVC, and now the problem is with the required redis instance, which should be up and running to finish the job.

paolomainardi avatar Jan 29 '23 16:01 paolomainardi

I tried running the migration job along with the other deployments, and it worked fine; it is the same approach the GitLab chart is taking. The concept is to let the scheduler restart the services until the migration job finishes to initialize the services; once finished, the pods start to come up and work fine.

Gitlab migration job used as a reference: https://docs.gitlab.com/charts/charts/gitlab/migrations/

paolomainardi avatar Jan 30 '23 16:01 paolomainardi

Thanks for your work on this @paolomainardi!

I tried running the migration job along with the other deployments, and it worked fine; it is the same approach the GitLab chart is taking. The concept is to let the scheduler restart the services until the migration job finishes to initialize the services; once finished, the pods start to come up and work fine.

How would this work for version upgrades? The pre-upgrade migrations needs to be run before any of the new version application pods, otherwise those can generate errors on some requests (when trying to access a table that has not been migrated yet), while their /health endpoint returns OK (it does not check the schema version).

I worry it will create user-facing errors during the migration, or even a server to become unavailable if the migration does not happen and all pods are upgraded to the new version.

renchap avatar Feb 17 '23 10:02 renchap

@renchap yes, you're right; the issue with this approach is that users can face problems while migrations are running.

This issue can only be overcome by just using Helm, and the best choice is to avoid running them as is doing this chart and move most of the complexity to the application side.

Always looking how Gitlab does, they open sourced the database migration types they support: https://docs.gitlab.com/ee/development/migration_style_guide.html

The case for Mastodon is "Regular migrations" which according to their document must be always under 3 minutes if higher than must be moved on post-deployment or background migrations.

Is not very clear indeed, what it happens during the 3-minutes window, maybe the migrations are always written in a way that prev/next releases are always compatible.

This is migration helm chart documentation: https://docs.gitlab.com/charts/charts/gitlab/migrations and from my direct experience, they run along the other deployments.

paolomainardi avatar Feb 27 '23 12:02 paolomainardi

is there any chance this could be moved forward?

jessebot avatar Jul 04 '23 10:07 jessebot