Long running celery-beat queries crash postgres database

[bioworkflows]

Summary:

The database backend of my website crashes today. Upon inspection, I found all slots are filled with database query, and ran more than 8 hrs.

SELECT "django_celery_beat_periodictasks"."ident", "django_celery_beat_periodictasks"."last_update" FROM 
"django_celery_beat_periodictasks" WHERE "django_celery_beat_periodictasks"."ident" = 1 LIMIT 21 FOR UPDATE

My periodic database had hundreds of insertion/deletion operations today but it has less than 50 entries at peak, so I am wondering what is going on here. Is it some sort of deadlock that prevented these queries from completion?

• Celery Version: celery==5.3.0b1
• Celery-Beat Version: django-celery-beat==2.5.0

Exact steps to reproduce the issue:

I am not sure how to reproduce this.

Detailed information

• Right now I have 25 periodic tasks, 20 or them are executed daily, 5 of them are on crontab, also run daily.
• The database was hosted on DigitalOcean with a connection pool. The following is the "Currently running queries" from the admin

[rjcampion3]

We've experienced this issue a few times now and it does eventually eat up all of the available connections and take down the database. Any updates on a possible root cause?

[curtisim0]

Hey at @bioworkflows or @rjcampion3 any work around that y'all came up with? I just upgraded a project and am having connections get slurped up by this.

[rjcampion3]

@curtisim0 We did the following settings in django:

CELERY_BROKER_TRANSPORT_OPTIONS = {
    "socket_keepalive": True,
    "socket_keepalive_options": {
        socket.TCP_KEEPIDLE: 60,
        socket.TCP_KEEPCNT: 5,
        socket.TCP_KEEPINTVL: 10,
    },
}

And we added --without-mingle to the celery command. That seemed to do the trick. I think the without-mingle was the main thing that seemed to resolve it, but the other settings are probably good to have anyway