warehouse icon indicating copy to clipboard operation
warehouse copied to clipboard

Set migration timeouts via environment variables

Open miketheman opened this issue 1 year ago • 0 comments

We occasionally must increase lock_timeout and statement_timeout for migrations that may take longer for highly-active tables.

Here's an example in our migration template: https://github.com/pypi/warehouse/blob/66b67309431e5726a82c867f74bbcd43a5b50bbf/warehouse/migrations/script.py.mako#L28-L48

The behavior here is often that once we see the failure, we:

  • either add or increase the values in the migration
  • commit, pull request, get approval, merge
  • cabotage tries with the new values

This creates a longer timeline than is probably needed, especially since this is not an exact science, more of a trial-and-find-out.

Another idea is to have these statements be set always, and use envvars to control them. Thus the process can become, once a failure is observed:

  • set/increase the environment variable values
  • create a new Release in cabotage

This could cut down the time needed to deploy with a new value, but the flip side is that I don't think there's a good way today to say to cabotage "if this succeeds, remove the envvar" to prevent misuse from the future deployment releases.

miketheman avatar Jul 08 '24 19:07 miketheman