Alert on time between when Celery task is added to queue and completes > N

[feoh]

Description/Context

The idea is to alert when the queue is backed up.

Plan/Design

I don't honestly know what's involved here or whether or not celery-monitoring is a good candidate for this work.

[rhysyngsun]

This issue on celery suggests adding the enqueued time to the request/headers: https://www.github.com/celery/celery/issues/5696

It's not implemented yet but there are some workaround solutions in the comments. If we setup that and then log the delta as the task completes that should let us graph/alert on it.