django-celery icon indicating copy to clipboard operation
django-celery copied to clipboard

beat and cam memory footprint reduction techniq (not an issue)

Open Sovetnikov opened this issue 8 years ago • 0 comments

When beat or celerycam in running as separate processes ("manage.py celery beat" or "manage.py celery celerycam" executed) all django application is initialized and all dependent stuff loaded in memory.

In my case, my django app takes about 70-120Mb private memory on start and because of it, on production servers, beat and cam processes take the same amount of memory as regular celery worker or uwsgi process. That's about 200Mb in my case for beat+cam.

I wrote two scripts that run beat and cam with minimum dependencies: djcelery as installed app, database connection from main app config. At now beat+cam processes take only 64Mb (by 32Mb on one process).

Maybe my results will be useful to someone with memory optimization.

celerybeat.py.txt celerycam.py.txt

Sovetnikov avatar Oct 24 '16 09:10 Sovetnikov