Celery-flower 0.9.2 using 100% CPU.

[mlunny-tabc]

celery-flower needed to be SIGKILLed after running at 100% CPU usage for 12+ hours.

Running against Celery 4.1.0 and RabbitMQ 4.1.0 on Ubuntu 14.04 LTS.

This is the first time we have seen the issue.

[codeadict]

I'm having this very same issue

[MilanPecov]

I'm having the very same issue

[redsymbol]

Hi, is there any progress on this issue? I'm consistently experiencing it just before I'm planning to deploy flower for the first time to production, for something mission-critical, and now I'm wondering if I should find another solution. Is there a workaround?

Thanks for this excellent piece of software. Sincerely, this issue notwithstanding, it's fantastic overall.

[redsymbol]

Update: I downgraded to 0.9.1, and am still experiencing the issue.

I'm so far running flower in a Vagrant environment (bento/ubuntu-18.04), backed by VirtualBox. None of the several other Python applications running on this vm are pegging the CPU, just flower; so I don't believe it has anything to do with the fact that it's running in this vm.

I'm noticing it specifically when using async-apply API calls (e.g., an http POST to /api/task/async-apply/tasks.some_task). This POST seems to trigger the CPU activity. Then, if I restart flower in supervisord (i.e. restart the flower process), I can then successfully make api calls for a while. Eventually a POST will trigger the high-cpu usage again.

I'm running flower in a virtual environment, with Python 3.6.7, if that matters.

I'm happy to dig into this more on my end, but I'm not familiar with the flower codebase, and could use some guidance. Any suggestions or ideas?

[AndreiPashkin]

Has anyone found any workarounds?

[KursLabIgor]

i have another issue on EC2. consume full disk space ~20Gb and instance stuck. have no idea.

[mher]

https://github.com/mher/flower/pull/1111 can cause the cpu increase. Please try the latest master version

[yarikoptic]

FWIW, we are experiencing high cpu usage with https://github.com/datalad/datalad-registry/ : please advise on what information could assist in troubleshooting/resolving it. Here is a py-spy top on it to possibly give

[candleindark]

We are currently at Flower version 2.0.1 and Celery version 5.3.1 and experiencing high CPU usage.

What we are experiencing in https://github.com/datalad/datalad-registry/ is that as the number of tasks being tracked grows the percentage of CPU usage grows.

1. At 440 tasks, 3.01% CPU usage
2. At 561 tasks, 3.59% CPU usage
3. At 1177 tasks, 7.57% CPU usage
4. At 6424 tasks, 22.74% CPU usage
5. At 20,000 tasks, 42.21% CPU usage
6. At 100,000 tasks (the default value of the max_tasks configuration option) , 80.59% CPU usage

As you can see, the CPU usage is too high even for the default max number of tasks to be tracked, and for our application, we want the number of tasks to be tracked to be even higher.

We are very happy with the features that Flower provides, it would be perfect if you can point us to a solution to this high CPU usage problem.

Thanks in advance.

[candleindark]

It seems to us that this problem originates from the repeated rewriting of the flower database file in persistent mode. By turning off the persistent mode, we are no longer experiencing this problem. For more detailed information, please visit https://github.com/datalad/datalad-registry/issues/301.

It would be nice if we can set Flower to persistent mode without experiencing this problem.