Celery Cron: Spurious missed_checkin issues

[rainsville]

How do you use Sentry?

Self-hosted/on-premise

Version

2.5.1

Steps to Reproduce

We have a Celery Redbeat setup with tasks that are scheduled to run every minute (using the interval scheduler). In the Sentry UI, we can see that Sentry expects the tasks to run at hh:mm:00. In practice, the tasks are started on time, but around the hh:mm:57 mark.

Occasionally, Sentry will report tasks as missing. We have noticed two different scenarios:

• the task is reported as started but in the "next minute" window (this is contrary to when we see the task started in the Celery logs)
• the task is reported as started at the correct time, but the task is still marked as missed in Sentry

Expected Result

The expectation is to not mark the tasks as missed. To account for potential delays in reporting the checkin to Sentry, it may be useful to allow specifying a checkin_margin for Celery tasks.

Actual Result

Example 1: task is expected to start at Jun 18, 12:27:00 PM UTC, is reported as started at Jun 18, 12:28:02 PM UTC and marked as Missed. In the Celery logs, the task is actually started at Jun 18, 12:27:57 PM UTC.

Example 2: task is expected to start at Jun 18, 12:37:00 PM UTC, starts at Jun 18, 12:37:56 PM UTC, but is still marked as Missed.

[sentrivana]

Hey @rainsville, thanks for writing in.

Could you add debug=True to your sentry_sdk.init and check whether you see an event being sent out for the missed check-ins? Trying to determine whether this is an issue between the SDK and Sentry or if the check-ins are not even being emitted by the SDK.

I'd also recommend you try keep_alive=True. This seemed to improve the situation for folks who reported similar behavior, so it might also help in your case.

[getsantry[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀