2.0.15 regression: bridge is not created, with "Expiring client local.xxx due to timeout"

[svet-b]

After upgrading to 2.0.15, we observed that in about 50% of cases broker bridges are not created on start-up. The log shows the message

Expiring client local.<bridge-name> due to timeout.

immediately on start-up (i.e. not after an actual conceivable timeout).

This was first reported in https://github.com/eclipse/mosquitto/issues/1334#issuecomment-1219606937 and subsequent comments in that thread (thanks @everactivetim, @vomov), though it's almost certainly a separate issue from #1334. It appears to have been an unintentional side effect of https://github.com/eclipse/mosquitto/commit/c99502a2567bf9147b3d811cb17e38ad51f08660. Indeed, the issue does not seem to occur in 2.0.14.

In our specific case this is on Linux on ARM.

[everactivetim]

We also reverted to 2.0.14 to solve our problem.

[gerardsyd]

Same - had this issue and glad I saw this as I was pulling my hair out. On 2.0.15, I was only getting these logs:

2022-09-14T21:17:29: Connecting bridge lollipop_mqtt (192.168.1.208:1883)
2022-09-14T21:17:29: Warning: Unable to connect to bridge lollipop_mqtt.
2022-09-14T21:17:29: mosquitto version 2.0.15 running
2022-09-14T21:17:29: Expiring client local.3410804944a7.lollipop_mqtt due to timeout.

[haizz]

Having the same problem on Windows machine:

1664897688: mosquitto version 2.0.15 starting
1664897688: Config loaded from C:\Program Files\mosquitto/mosquitto.conf.
1664897688: Opening ipv4 listen socket on port 1883.
1664897688: Bridge local.PRODUCTION-MOSQ.main doing local SUBSCRIBE on topic ets/sensors/#
1664897688: Bridge local.PRODUCTION-MOSQ.main doing local SUBSCRIBE on topic test/#
1664897688: Connecting bridge main (mqtt:8883)
1664897688: Bridge PRODUCTION-MOSQ.main sending CONNECT
1664897688: mosquitto version 2.0.15 running
1664897688: Expiring client local.PRODUCTION-MOSQ.main due to timeout.

[jld-adriano]

Same issue

1666866838: Config loaded from /mosquitto/config/mosquitto.conf.
1666866838: Opening ipv4 listen socket on port 1886.
1666866838: Opening ipv6 listen socket on port 1886.
1666866838: Connecting bridge mosquitto-device-5 (xxx)
1666866838: mosquitto version 2.0.15 running
1666866839: Expiring client local.mosquitto-device-5 due to timeout.
1666866839: New connection from ::1:51704 on port 1886.
1666866839: New connection from ::1:51706 on port 1886.
1666866839: New client connected from ::1:51704 as sparkplug-maestro (p2, c1, k3000).
1666866839: New client connected from ::1:51706 as old-maestro (p2, c1, k60).
1666866841: New connection from 127.0.0.1:40616 on port 1886.
1666866841: New client connected from 127.0.0.1:40616 as processor (p2, c1, k20).```

[karlp]

I'm pretty sure I was seeing this as well today. That "expiring client local.blah due to timeout", and with zero extra info available. In my case I was only bridging to a same network, 192.168.blah, no encryption, and restarting fixed things, couldn't figure it out. I'll hopefully be able to collect more information though.

[jld-adriano]

In my case it happens every 2nd reboot, on the clock. Reboot -> expires, reboot -> works , reboot -> expires, etc...

[markdegrootnl]

How is this still not fixed?

[Nightenom]

Can confirm with docker v2.0.15 on Debian 11 armv7l, revert to 2.0.14 fixes the problem

[serroba]

Can report the same thing with: macOS Ventura 13 M1 docker 20.10.18 v2.0.15

It was intermittent in my case. Not consistent when I could/couldn't not have the bridge running.

[pod67]

We upgraded to 2.0.15 due to https://github.com/eclipse/mosquitto/issues/2594 but now we have to downgrade due to this bug.

This bug is severe, please look into this.

[lepgithub]

Same issue with 2.0.15. It appears the issue has been known for some time; why in Jan 2023 it's still not fixed?

Jan 15 13:12:51 micro mosquitto.mosquitto[1597]: 1673817171: Connecting bridge (step 2) 2cloudww (cloud.x.com:8883) Jan 15 13:12:51 micro mosquitto.mosquitto[1597]: 1673817171: Expiring client local.micro.2cloudww due to timeout.

[pod67]

Same issue with 2.0.15. It appears the issue has been known for some time; why in Jan 2023 it's still not fixed?

Jan 15 13:12:51 micro mosquitto.mosquitto[1597]: 1673817171: Connecting bridge (step 2) 2cloudww (cloud.x.com:8883) Jan 15 13:12:51 micro mosquitto.mosquitto[1597]: 1673817171: Expiring client local.micro.2cloudww due to timeout.

@ralight ??

[krzakx]

Hey. I do face similar issue on Hassio. With Mosquitto broker Add-On version: 6.2.0

[EpicLPer]

Same issue here right now trying to connect to HiveMQ, and it doesn't reconnect anymore as it says in the docs. I made a separate issue for this here: #2795

[ghost]

Same issue here: mosquitto bridge 2.0.15 to mosquitto. I had to downgrade to 2.0.14 and now is working good

[dj0abr]

same problem here, how to downgrade to 2.0.14 on Linux, I did not find a package

[dj0abr]

reboot does not solve this problem

[kzajac83]

https://github.com/home-assistant/addons/issues/3072

I create above issue, today found that root of cause is timeout to bridge broker. I'm wiaitng for fix but to workaround how to increase timeout for bridge and what value is default?

[kelvincesar]

Same problem here. I'm using version 2.0.15, and the issue occurs every second start of mosquitto.

I'm using the following settings on bridge:

connection test
cleansession false
try_private false
bridge_attempt_unsubscribe false
notifications true
max_inflight_messages 5
persistence true
persistence_location /var/lib/mosquitto
autosave_interval 60
persistent_client_expiration 30d
bridge_capath /etc/ssl/certs

[Makin-Things]

@ralight can you provide any insight/update on this issue?

[nikito7]

still there...

persistence false "fixes" but create another issues

[swiergot]

I have submitted a PR.

Please note that I had zero experience with mosquitto code prior to this so if you decide to use the fix before it is reviewed by maintainers, you risk running into potential side effects I have not foreseen.

[swiergot]

Nevermind, apparently the fix (https://github.com/eclipse/mosquitto/commit/6d240a9d183413fca7d2a4f42eae803ba84af97c) has been on the fixes branch for over 3 months. There just hasn't been any new release since.

[nikito7]

In my case it happens every 2nd reboot, on the clock. Reboot -> expires, reboot -> works , reboot -> expires, etc...

I confirm this.

But no need reboot.

Just stop and start mosquitto

[tschweitzer]

Wondering if there is any progress on this since requires me a manual restart every day?

[chennin]

Yes, it's fixed but not released. You can:

• Revert to .14
• Restart it manually
• Wait for a new release

[kzajac83]

When is planned new release?

[EpicLPer]

Awesome! Thanks @ralight, almost thought Mosquitto was dead :)

[svet-b]

I guess it's on an annual release schedule now :)

2.0.16 - 2023-08-16 2.0.15 - 2022-08-16