'socket_domain ipv4' breaks config (listen twice -> Address already in use)

[antoine-sac]

I have been troubleshooting a bug preventing mosquitto from booting because it attempts to listen twice to the same port. I finally tracked it down to the socket_domain option.

Here is the simplest reproducible example which fails to start.

cat /etc/mosquitto/test.conf

listener 1885 localhost
socket_domain ipv4

mosquitto -c /etc/mosquitto/test.conf -v

1719099713: mosquitto version 2.0.18 starting
1719099713: Config loaded from /etc/mosquitto/test.conf.
1719099713: Opening ipv4 listen socket on port 1885.
1719099713: Opening ipv4 listen socket on port 1885.
1719099713: Error: Address already in use

And no, I definitely do NOT have another instance running. I can reproduce the issue on any random port.

1719100410: mosquitto version 2.0.18 starting
1719100410: Config loaded from /etc/mosquitto/test.conf.
1719100410: Opening ipv4 listen socket on port 12345.
1719100410: Opening ipv4 listen socket on port 12345.
1719100410: Error: Address already in use

If I remove the "socket_domain" option, I get the expected behaviour:

mosquitto -c /etc/mosquitto/test.conf -v
1719100163: mosquitto version 2.0.18 starting
1719100163: Config loaded from /etc/mosquitto/test.conf.
1719100163: Opening ipv6 listen socket on port 1885.
1719100163: Opening ipv4 listen socket on port 1885.
1719100163: mosquitto version 2.0.18 running

[Daedaluz]

@ralight This is a bit strange. My c might be rusty, but the getaddrinfo here

https://github.com/eclipse/mosquitto/blob/9f69df493dca6acf3b181927b2a691eaf2b059db/src/net.c#L719

seems to return two identical entries when socket_domain is set.

I don't remember this was a thing....

[ralight]

@antoine-sac I was able to reproduce this by editing my /etc/hosts file to have a duplicate entry:

127.0.0.1 localhost
127.0.0.1 localhost

Is this the case for you as well?

[cloudbells]

Also experiencing this on 2.0.18.

Here is the full config that causes the bug:

persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
log_dest stdout
log_type all
log_timestamp_format [%Y-%m-%d %H:%M:%S]
socket_domain ipv4
listener 1883

## Authentication ##
allow_anonymous false
password_file /mosquitto/config/password.txt

Logs:

mosquitto      | [2024-10-07 15:34:14]: mosquitto version 2.0.18 starting
mosquitto      | [2024-10-07 15:34:14]: Config loaded from /mosquitto/config/mosquitto.conf.
mosquitto      | [2024-10-07 15:34:14]: Opening ipv4 listen socket on port 1883.
mosquitto      | [2024-10-07 15:34:14]: Opening ipv6 listen socket on port 1883.
mosquitto      | [2024-10-07 15:34:14]: Opening ipv4 listen socket on port 1883.
mosquitto      | [2024-10-07 15:34:14]: Error: Address in use
mosquitto exited with code 1

After switching the order of the listener and socket_domain entries, it works as expected:

persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
log_dest stdout
log_type all
log_timestamp_format [%Y-%m-%d %H:%M:%S]
listener 1883
socket_domain ipv4

## Authentication ##
allow_anonymous false
password_file /mosquitto/config/password.txt

Success logs:

mosquitto      | [2024-10-07 15:40:32]: mosquitto version 2.0.18 starting
mosquitto      | [2024-10-07 15:40:32]: Config loaded from /mosquitto/config/mosquitto.conf.
mosquitto      | [2024-10-07 15:40:32]: Opening ipv4 listen socket on port 1883.
mosquitto      | [2024-10-07 15:40:32]: mosquitto version 2.0.18 running

[Daedaluz]

Having the socket domain option above the first listener probably activates the default listener though.

[cloudbells]

Not sure, but changing listener port in the config works when they're swapped like that at least.

[ralight]

@cloudbells The answer by @Daedaluz is correct here. What you effectively have is:

listener 1883 # implicit listener caused by socket_domain without a listener
socket_domain ipv4
listener 1883

[Daedaluz]

@ralight I can't seem to figure out why it would return two identical entries on my system (I do not have double localhost entries in the hosts file), but i think it's safe to say that it depends on the system environment at least.

Pop_OS! / Systemd seem to have some resolver running and point /etc/resolv.conf to it. The ip in there is some local address i do not recognize and doesn't list with ip -a (perhaps it's in some network namespace?)

Do you think this is something worth "fixing" in mosquitto or not? The whole issue is very subtle, yet annoying if you hit it.

Edit: now when i wrote it down, perhaps it's the fact that there is a dns server that resolve localhost to 127.0.0.1 AND a 127.0.0.1 in the hosts file.

[ralight]

@Daedaluz From what I've seen, this is possibly a misconfiguration outside of mosquitto that needs fixing. That's the assumption I'm working under, and if it is the case then I don't think it's a problem mosquitto should be fixing.

[Daedaluz]

Hah...

I think we just stumbled on a glibc bug:

• https://sourceware.org/bugzilla/show_bug.cgi?id=14969
• https://bugzilla.redhat.com/show_bug.cgi?id=496300
• https://sourceware.org/bugzilla/show_bug.cgi?id=4980

Apparently, it has to do with ::1 localhost beeing treated as an ipv4 mapped ipv6 address.

[ralight]

I guess I was half right then. I guess we can only say for sure if @antoine-sac confirms.

[antoine-sac]

Hi all, apologies for the delay, indeed removing the ipv6 localhost lines from /etc/hosts fixed the problem.

So it really is an external bug (the ::1 statement being treated as ipv4).

Thanks for your help.