warehouse
warehouse copied to clipboard
Set migration timeouts via environment variables
We occasionally must increase lock_timeout and statement_timeout for migrations that may take longer for highly-active tables.
Here's an example in our migration template: https://github.com/pypi/warehouse/blob/66b67309431e5726a82c867f74bbcd43a5b50bbf/warehouse/migrations/script.py.mako#L28-L48
The behavior here is often that once we see the failure, we:
- either add or increase the values in the migration
- commit, pull request, get approval, merge
- cabotage tries with the new values
This creates a longer timeline than is probably needed, especially since this is not an exact science, more of a trial-and-find-out.
Another idea is to have these statements be set always, and use envvars to control them. Thus the process can become, once a failure is observed:
- set/increase the environment variable values
- create a new Release in cabotage
This could cut down the time needed to deploy with a new value, but the flip side is that I don't think there's a good way today to say to cabotage "if this succeeds, remove the envvar" to prevent misuse from the future deployment releases.