flower Rabbitmq broker API times out

Describe the bug When using rabbitmq as the broker and specifying broker_api for Flower to use the request for api/queues times out with:

[E 210605 19:25:16 broker:78] RabbitMQ management API call failed: Timeout while connecting

Using identical url from the browser to reach the api/queues endpoint of rabbitmq management api is successful.

To Reproduce Steps to reproduce the behavior:

Launch flower based on newest master with: celery -A tasks -broker=amqp://guest:guest@localhost:5672// flower --debug --broker_api=http://guest:guest@localhost:15672/api/
Wait for flower to fully boot.
Start rabbitmq and wait for it to fully boot.
Go to localhost:5555
Click on broker tab, notice the spinning browser's connection wheel
Observe Flower's logs - eventually it will show the timeout error as above.

Similar situation happens if flower is started first and redis after.

Expected behavior The rabbitmq's api is responding correctly and Flower is able to use it.

Screenshots If applicable, add screenshots to help explain your problem.

System information Output of python -c 'from flower.utils import bugreport; print(bugreport())' command

flower   -> flower:1.0.1-dev tornado:6.1 humanize:3.3.0
software -> celery:5.0.5 (singularity) kombu:5.0.2 py:3.9.2
            billiard:3.6.3.0 py-amqp:5.0.5
platform -> system:Linux arch:64bit, ELF
            kernel version:5.4.0-70-generic imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled

deprecated_settings: None

Jun 05 '21 18:06 Tomasz-Kluczkowski

@mher I am looking into this one as it is I think the last remaining issue to fix and get new release so please assign me. Also it was reported here: https://stackoverflow.com/questions/66815640/how-can-i-connect-flower-to-the-broker-rabbitmq-in-docker-compose

Jun 05 '21 18:06 Tomasz-Kluczkowski

Results of investigation:

if rabbitmq (or broker in general) boots first (and is ready) and flower boots after, all is good (tested in docker-compose as well as launching rabbitmq/flower manually)
if flower boots first and we try to access broker tab, we get the timeout, unfortunately if we boot the rabbitmq now, even if it is fully booted flower will still fail to connect to it. Other tabs show no updated content (but no errors are logged when using them).

Tested if similar happens for Redis and it does although the broker tab does not result in spinning wheel of death and no Timeout is logged - still the dashboard, tasks tabs show no content and broker tab shows no real content.

Potential solutions:

find out if we can make Flower self-heal - have some mechanism of rebuilding the connection to broker?
how would this be done? It should be universal and one method somewhere called always as an attempt to retry.
do we have to check for connection to the broker to be present before making any calls? Probably not - just have a retry mechanism or make it a command line option?
for docker compose use wait-for-it script and start flower only after broker is ready (a copout really but what can one do)?
I will also fix a few bits in docker compose that are no longer valid (mainly using ENTRYPOINT ["flower"] as we can no longer do that in new version - have to use CMD as it can be fully overridden and ENTRYPOINT cannot).

@mher Can you point me at a place in the code where this issue takes place if you remember :)? Also when I boot Flower and there is no broker running, would you expect an exception being raised or an error logged in flower? Is there code that already checks that and stopped working?

Jun 05 '21 20:06 Tomasz-Kluczkowski

This is fixed in #1113 by making a connection check when booting.

Jun 20 '21 19:06 Tomasz-Kluczkowski

https://github.com/mher/flower/commit/d4ea1dcba6c42098fb5bb6e29bd82eeb8b6c2224 fixes the issue

Jun 20 '21 20:06 mher

Thx @mher, I will test in our production if it all works nicely. I think then I will convert my #1113 to just docs and docker compose fix.

Jun 20 '21 20:06 Tomasz-Kluczkowski

Thanks @Tomasz-Kluczkowski , you can keep tests as well.

Jun 20 '21 21:06 mher

I'm commenting just to say, as a fellow open source developer, that I'm impressed by you guys going so smoothly between two implementation variations - as well as contributing it on a Sunday.

So yeah, people are indeed watching (and loving) your work :)

Jun 20 '21 21:06 petarmaric

@mher I have tested your change (the added timeouts) and while yes, we no longer get the spinning wheel of death in the browser Flower still is no longer capable of connecting to the broker if it was booted without the broker running.

I have tested with rabbitmq and redis as brokers.

Test:

boot flower
go to localhost:5555
check tabs before broker started - no more super long timeouts for rabbitmq which is nice (redis never had this issue, but I checked it still does not have it :P).
boot broker and wait for it to fully boot
The broker tab seems to refresh for redis (shows the default celery queue)
The broker tab does not refresh for rabbitmq (all remains empty).
This is logged any time we try to access the broker tab in rabbitmq case (but the timeout now is 1s not forever :) ):

[E 210622 21:01:06 broker:79] RabbitMQ management API call failed: Timeout while connecting
[I 210622 21:01:07 web:2239] 304 GET /broker (::1) 1025.09ms

for both brokers the dashboard and tasks tab now show no content and never update.

If broker is started first and then Flower all is good.

I think we may need to merge my changes from #1113 with waiting for available broker as an improvement of sorts. In the end Celery itself does it and will not start if there is no broker. This is very important in kubernetes/docker deployment where we cannot guarantee order of booting of pods/containers to start Flower when the broker is working.

Please let me know what you think.

Jun 22 '21 20:06 Tomasz-Kluczkowski

I will check one more thing and report back.

Jun 22 '21 20:06 Tomasz-Kluczkowski

I tried and did not achieve anything @mher , I think making a check for broker connection on Flower's boot is the best option for now.

Please let me know if you want to merge it as it looks like I need to fix merge conflicts there.

Jun 22 '21 20:06 Tomasz-Kluczkowski

I'm not able to reproduce. Running without RabbitMQ, then starting, stoping RabbitMQ works as expected. Broker tab shows celery queue when RabbitMQ is available.

[D 210623 21:22:07 inspector:36] Sending stats inspect command
[D 210623 21:22:07 inspector:36] Sending active_queues inspect command
[D 210623 21:22:07 inspector:36] Sending registered inspect command
[D 210623 21:22:07 inspector:36] Sending scheduled inspect command
[D 210623 21:22:07 inspector:36] Sending active inspect command
[D 210623 21:22:07 inspector:36] Sending reserved inspect command
[D 210623 21:22:07 inspector:36] Sending revoked inspect command
[D 210623 21:22:07 inspector:36] Sending conf inspect command
[I 210623 21:22:14 web:2239] 304 GET /dashboard (::1) 42.40ms
[I 210623 21:22:15 web:2239] 200 GET /dashboard?json=1&_=1624497735046 (::1) 1.34ms
[I 210623 21:22:15 web:2239] 304 GET /tasks (::1) 18.24ms
[I 210623 21:22:16 web:2239] 200 POST /tasks/datatable (::1) 2.98ms
[E 210623 21:22:17 broker:79] RabbitMQ management API call failed: Timeout while connecting
[I 210623 21:22:17 web:2239] 200 GET /broker (::1) 1033.35ms
[D 210623 21:22:49 channel:104] using channel_id: 1
[D 210623 21:22:49 channel:444] Channel open
[D 210623 21:22:49 events:175] Capturing events...
[I 210623 21:22:49 mixins:226] Connected to amqp://guest:**@127.0.0.1:5672//
[D 210623 21:22:49 channel:104] using channel_id: 1
[D 210623 21:22:49 channel:444] Channel open
[D 210623 21:22:50 connection:726] heartbeat_tick : for connection fa534377364f4406a3347da30bb814dc
[D 210623 21:22:50 inspector:39] Inspect command reserved took 43.31s to complete
[W 210623 21:22:50 inspector:42] Inspect method reserved failed
[D 210623 21:22:50 inspector:39] Inspect command stats took 43.32s to complete
[W 210623 21:22:50 inspector:42] Inspect method stats failed
[D 210623 21:22:51 inspector:39] Inspect command active took 43.62s to complete
[W 210623 21:22:51 inspector:42] Inspect method active failed
[D 210623 21:22:51 inspector:39] Inspect command scheduled took 43.63s to complete
[D 210623 21:22:51 inspector:39] Inspect command revoked took 43.63s to complete
[W 210623 21:22:51 inspector:42] Inspect method scheduled failed
[D 210623 21:22:51 inspector:39] Inspect command conf took 43.62s to complete
[D 210623 21:22:51 inspector:39] Inspect command registered took 43.63s to complete
[W 210623 21:22:51 inspector:42] Inspect method revoked failed
[W 210623 21:22:51 inspector:42] Inspect method conf failed
[W 210623 21:22:51 inspector:42] Inspect method registered failed
[D 210623 21:22:51 inspector:39] Inspect command active_queues took 43.64s to complete
[W 210623 21:22:51 inspector:42] Inspect method active_queues failed
[D 210623 21:22:51 connection:726] heartbeat_tick : for connection fa534377364f4406a3347da30bb814dc
[I 210623 21:23:07 web:2239] 200 GET /broker (::1) 78.92ms
[D 210623 21:23:09 connection:726] heartbeat_tick : for connection fa534377364f4406a3347da30bb814dc
[I 210623 21:23:09 web:2239] 304 GET /tasks (::1) 20.86ms
[I 210623 21:23:09 web:2239] 200 POST /tasks/datatable (::1) 4.07ms
[I 210623 21:23:10 web:2239] 200 GET /dashboard (::1) 26.37ms
[I 210623 21:23:10 web:2239] 200 GET /dashboard?json=1&_=1624497790900 (::1) 0.59ms
[D 210623 21:23:11 connection:726] heartbeat_tick : for connection fa534377364f4406a3347da30bb814dc
[I 210623 21:23:11 web:2239] 304 GET /tasks (::1) 23.41ms
[I 210623 21:23:11 web:2239] 200 POST /tasks/datatable (::1) 1.80ms
[D 210623 21:23:12 connection:726] heartbeat_tick : for connection fa534377364f4406a3347da30bb814dc
[D 210623 21:23:13 connection:726] heartbeat_tick : for connection fa534377364f4406a3347da30bb814dc
\[D 210623 21:23:16 channel:147] Closed channel #1
[E 210623 21:23:16 events:184] Failed to capture events: '(0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'', trying again in 1 seconds.
[D 210623 21:23:16 events:187] (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
[E 210623 21:23:34 broker:79] RabbitMQ management API call failed: [Errno 61] Connection refused
[I 210623 21:23:34 web:2239] 200 GET /broker (::1) 50.91ms
[I 210623 21:23:35 web:2239] 304 GET /tasks (::1) 20.51ms
[I 210623 21:23:35 web:2239] 200 POST /tasks/datatable (::1) 2.62ms
[D 210623 21:24:05 channel:104] using channel_id: 1
[D 210623 21:24:05 channel:444] Channel open
[D 210623 21:24:15 connection:726] heartbeat_tick : for connection c17425d70afa424d87d4422f55308906
[D 210623 21:24:17 connection:726] heartbeat_tick : for connection c17425d70afa424d87d4422f55308906
[I 210623 21:24:23 web:2239] 200 GET /broker (::1) 61.83ms
[I 210623 21:24:24 web:2239] 304 GET /tasks (::1) 20.35ms
[I 210623 21:24:24 web:2239] 200 POST /tasks/datatable (::1) 4.48ms
[D 210623 21:24:25 connection:726] heartbeat_tick : for connection c17425d70afa424d87d4422f55308906

Jun 24 '21 01:06 mher

I have to check if it makes the difference but my testing was done solely with rabbitmq/redis in docker containers.

This is to simulate closer to our production where it runs in a kubernetes pod.

Are you running rabbitmq for this test inside of a container too?

If not maybe that's the difference that makes or breaks Flower.

If that is the case I think a lot use cases are in containerised environments nowadays and Flower should cater to that too.

I will investigate more tomorrow evening.

Jun 25 '21 21:06 Tomasz-Kluczkowski

Are you running rabbitmq for this test inside of a container too?

Yes

Jun 25 '21 22:06 mher

Hello, I just did extensive tests and still cannot make it connect to broker if rabbitmq in docker is started after Flower is finished booting up.

I tried initially using pycharm configuration to start the rabbitmq , then deleted the container/image and begun starting rabbitmq via command line with:

docker run --name rabbitmq -d -p 5762:5672 -p 15762:15762 -p 15672:15672 --network host rabbitmq

and I start flower not in a container which maybe is the reason for all of this trouble? I just start it via:

celery --broker=amqp://guest:guest@localhost:5672// flower --debug --broker_api=http://guest:guest@localhost:15672/api/

Are you running Flower in a docker container for this test too @mher?

Is the latest Flower image based on latest master so I can test too, or have to build it myself?