High CPU Usage

[kedare]

Hello.

I am getting a quite high CPU usage from the celery monitor process, around 15% constantly, I'm not having a big load or queue so I can't really explain that (I process maybe like maximum 5 tasks per seconds in the worst case).

Do you have the same behaviour ? How can I troubleshoot that more ?

Here is my monitor command

pipenv run celery -A website_manager_service events -l info --camera django_celery_monitor.camera.Camera --frequency=2.0 --pidfile=

Thanks

[jezdez]

I have not seen this personally, but I'd be interested to know what version of Python, Celery, which broker and what result backend. Is the load reduced if you increase the frequency of the camera?

[kedare]

Same load with either 0.5 and 2.0 of interval. My result backend is the database.

My versions are Django==1.10.8 django-celery-monitor==1.1.2 celery==4.1.0

I didn't set any result backend, my config is like this (+ queue routing)

# Task scheduler
CELERY_BROKER_URL = os.getenv('BROKER_URL', 'amqp://guest@rabbitmq:5672/')
CELERY_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1

Using PostgreSQL as database backend. Everything is working fine but I'm just surprised of the CPU consumption thing (Also I can see there is a lot of network IO compared to the rest, looks like it's traffic to/from the database as the same IO appears on the SQL proxy)

There you have a screenshot to compare (You can see the IDLE workers and the celery monitor is the monitor one), I've set it the same CPU allocation than the workers to have the correct % cpu scale.

[kedare]

I confirm it looks like it's doing a lot of database queries. There you have a screenshot of pgAdmin4, all those small spikes are from celery monitor (When I stop it they disappear), is that normal ? That looks like a lot of queries (For a very few or no tasks running)