readthedocs.org
readthedocs.org copied to clipboard
Build: handle Celery timeout events
Solve the problem reported by Sentry when the Celery task times out: https://sentry.io/organizations/read-the-docs/issues/3038754375/
Each commit has a detailed explanation of the changes
Note to myself: the latest commit tries to handle soft timeout cancelled properly. It's close but it needs a little more of QA because for some reason SIGUSR1
is triggered/catched as well.
Note that we are not hitting this too frequently because we have an pretty high timeout due to this: https://github.com/readthedocs/readthedocs.org/blob/634f85bf00ffcab7bc91926cc9bf9cc394d375fb/readthedocs/projects/tasks/utils.py#L95-L100
I'm closing this PR for now. We can come back in the future if we consider. The idea is good, but I wasn't able to properly QA on production to be sure it works as I expect.