Support for celery

[ehiggs]

As a developer working with distributed systems, I use a task queue to make some work asynchronous so that long running tasks do not block the web server threads. To profile the workers, we run the worker system using

celery worker -Q queuename

I would like to run

scalene --reduced-profile --html --outfile celery-profile.html celery worker -Q queuename

This 'works', but the result is a html file that has the celery main function highlighted and listed as using all the memory and cpu.

Celery has a few ways of managing workers.

1. prefork. This uses multiprocessing to spawn processes to do the work.
2. gevent. This converts workers to use gevent green threads.
3. eventlet. Similar to gevent.

I tested scalene with prefork.

[emeryberger]

Can you share the celery-profile.html file? If not, a small example that reproduces the problem would be immensely helpful - thanks!

[ehiggs]

I've made a small example project:

https://github.com/ehiggs/scalene-celery-demo

[ehiggs]

Hi, is there anything I can do to unblock investigation here? Thanks!

[ducminhle]

Did you find a solution for this issue, @ehiggs?

I try to run scalene with celery but /tmp/out.json content is {}, @emeryberger.

python -m scalene --cli --profile-interval 1.0 --json --outfile /tmp/out.json /usr/local/bin/celery -A director._auto:cel worker --concurrency=1 --pool=prefork --prefetch-multiplier=1 --loglevel=INFO --queues="abc"

[ehiggs]

No the closest solution I found here was to write some tests for the application level code and use scalene on that. But I never got it working with celery itself.