Failed migration on update 2022-06-01 post-mortem

[mattyg]

Failed Deployment Post-Mortem

• In #397 we created a migration to set reasonRealized for all existing praise -- by replacing discord's machine-readable formatting of users and channels with human-readable formatting
• The migration failed when it did not receive all successful responses from the discord api. (see image below)
• Because of this, the migration was replaced with another that set reasonRealized for all existing praise to praise.reason.

Log of failed migraiton

Research

• Migration failed on discord api request to fetch info on a channel 948663379487252530
• The channel is a thread - Omega > Consilience library
• Unable to run /praise command inside a thread currently (see image below)

Reflections

In the future we might want to change how we handle failed migrations:

• When a migration fails on deployment, revert to the previous deployment. Currently the process would be:

  1. Revert to previous codebase
  2. Run all migrations in reverse yarn run migrator down (for each new migration)

• Additionally, it might be easier if we squashed multiple migrations into one, for any release that has multiple migrations. That way, the process to revert to a previous deployment would be:

  1. Revert to previous codebase (the database would already be unchanged due to the failed migration).

[Vyvy-vi]

Some workarounds to fix the migrations -

1. Using the most openly accessible channel in the server when calling Util.cleanContent(): This ensures that the bot never has to query the channel in which praise was sent, resolving the issue we're having above. And. this also allows us to realise the content better by ensuring that all user ids get rendered irrespective of them having access to that channel. OR
2. Write code to fetch the channels that can account for different types of channels returned used in discord.js. OR
3. Re-write Util.cleanContent and use our own implementation that internally queries the data from the Guild object.