django-health-check
django-health-check copied to clipboard
Celery health check timeout exceptions and TaskRevokedError
When I use the celery check, I will get some errors every 2-3 minutes. there are two kinds of errors like below. However my celery runs well without a problem.
2023-10-07T00:02:53.476107594Z unavailable: TaskRevokedError: The task was revoked, likely because it spent too long in the queue
2023-10-07T00:02:53.476143094Z Traceback (most recent call last):
2023-10-07T00:02:53.476149594Z File "/usr/local/lib/python3.9/site-packages/health_check/contrib/celery/backends.py", line 20, in check_status
2023-10-07T00:02:53.476164894Z result.get(timeout=result_timeout)
2023-10-07T00:02:53.476168894Z File "/usr/local/lib/python3.9/site-packages/celery/result.py", line 250, in get
2023-10-07T00:02:53.476172794Z return self.backend.wait_for_pending(
2023-10-07T00:02:53.476176494Z File "/usr/local/lib/python3.9/site-packages/celery/backends/base.py", line 763, in wait_for_pending
2023-10-07T00:02:53.476180394Z return result.maybe_throw(propagate=propagate, callback=callback)
2023-10-07T00:02:53.476184194Z File "/usr/local/lib/python3.9/site-packages/celery/result.py", line 364, in maybe_throw
2023-10-07T00:02:53.476188094Z self.throw(value, self._to_remote_traceback(tb))
2023-10-07T00:02:53.476191695Z File "/usr/local/lib/python3.9/site-packages/celery/result.py", line 357, in throw
2023-10-07T00:02:53.476195595Z self.on_ready.throw(*args, **kwargs)
2023-10-07T00:02:53.476199495Z File "/usr/local/lib/python3.9/site-packages/vine/promises.py", line 234, in throw
2023-10-07T00:02:53.476203395Z reraise(type(exc), exc, tb)
2023-10-07T00:02:53.476206995Z File "/usr/local/lib/python3.9/site-packages/vine/utils.py", line 30, in reraise
2023-10-07T00:02:53.476210795Z raise value
2023-10-07T00:02:53.476214395Z celery.exceptions.TaskRevokedError: expired
and
2023-10-07T00:43:51.514487089Z unavailable: TimeoutError: The task took too long to return a result
2023-10-07T00:43:51.514537089Z Traceback (most recent call last):
2023-10-07T00:43:51.514543189Z File "/usr/local/lib/python3.9/site-packages/health_check/contrib/celery/backends.py", line 20, in check_status
2023-10-07T00:43:51.514547189Z result.get(timeout=result_timeout)
2023-10-07T00:43:51.514558590Z File "/usr/local/lib/python3.9/site-packages/celery/result.py", line 250, in get
2023-10-07T00:43:51.514563590Z return self.backend.wait_for_pending(
2023-10-07T00:43:51.514567390Z File "/usr/local/lib/python3.9/site-packages/celery/backends/base.py", line 755, in wait_for_pending
2023-10-07T00:43:51.514571290Z meta = self.wait_for(
2023-10-07T00:43:51.514574890Z File "/usr/local/lib/python3.9/site-packages/celery/backends/base.py", line 791, in wait_for
2023-10-07T00:43:51.514588490Z raise TimeoutError('The operation timed out.')
2023-10-07T00:43:51.514592190Z celery.exceptions.TimeoutError: The operation timed out.
Is it expected or I am missing something? Thanks!
We have the same problem:
unavailable: TaskRevokedError: The task was revoked, likely because it spent too long in the queue
Although the issue is happening randomly, the service and the connection still work properly.
Do we know why that is happening suddenly? Is there any workaround we can use?
@miaoz2001 @raulgzm did you guys ever figure this out?