Breadcrumbs accumulating for celery workers

[wojtekbauman]

Breadcrumbs for errors during celery tasks execution accumulate over time. It seems that each celery task error contains breadcrumbs for this particular celery worker since the beginning of its existence. The following screenshot shows this scenario:

As a work-around, we manually clear breadcrumbs once each task is finished:

@signals.task_postrun.connect
def on_task_postrun(**_):
    _clear_sentry_breadcrumbs()


@signals.task_internal_error.connect
def on_task_internal_error(**_):
    _clear_sentry_breadcrumbs()


@signals.task_unknown.connect
def on_task_unknown(**_):
    _clear_sentry_breadcrumbs()


def _clear_sentry_breadcrumbs():
    with sentry_sdk.configure_scope() as scope:
        scope.clear_breadcrumbs()

But maybe it would be better if the breadcrumbs and/or the whole sentry scope got cleared automatically?

[chadwhitacre]

What's the word here, @getsentry/team-webplatform? Ideas on how to address this?

[rhcarvalho]

We do have code that clears breadcrumbs in the Celery integration.

https://github.com/getsentry/sentry-python/blob/51987c57157102bbd32e1e7b084c26f4dc475d86/sentry_sdk/integrations/celery.py#L142

@wojtekbauman would you have a minimal example to reproduce the problem you're seeing?

Perhaps there's something we're missing out in the integration, needs more investigation.

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[antonpirker]

Hello @wojtekbauman Thanks for reporting this! This definitely looks like a bug. We plan to make a update for supporting Django 4 and I think this could also be looked at during this process.

[antonpirker]

Hey @wojtekbauman !

Better late than never. I just tried to reproduce this with Sentry SDK version 1.35.0, Python 3.9.15, celery 5.3.5 and redis as the Celery broker. Redis server version 7.2.1

I started a Celery preforked worker and then run the same task multiple times. The task first creates a breadcrumb with logging.info("the breadcrumb") and then triggers an error with logging.error("something").

All the tasks where run in the same worker and all the error events in Sentry only had one breadcrumb.

Could you try to reproduce this with the updated versions of the above mentioned libs or give me a code snippet where I can reproduce this? Without this there is nothing I can do I am afraid. Thanks!

[getsantry[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀