Random Socket error on client <unknown>, disconnecting

[joe-agent]

I have an Ubuntu 20.04 host running mosquitto container using eclipse-mosquitto:1.6.7

In the same host, I have a backend container with multiple C# microservices connecting to the mosquitto container. Those microservices use MQTTnet.Extensions.ManagedClient version 3.0.16

Over 1 week period, I notice socker error on client in the mosquitto logging like below:

2022-09-16T11:28:09.096130200Z 1663327689: New client connected from 172.18.0.10 as MbServer (p2, c1, k60).
2022-09-16T11:30:30.327170700Z 1663327830: New connection from 172.18.0.26 on port 1883.
2022-09-16T11:30:30.493826100Z 1663327830: New client connected from 172.18.0.26 as BackupRestore-ff96fd35-56ec-4342-a672-e7d6a819fafb (p2, c1, k15).
2022-09-16T11:30:45.165483800Z 1663327845: New connection from 172.18.0.26 on port 1883.
2022-09-16T11:30:45.254329700Z 1663327845: New client connected from 172.18.0.26 as HartIP-44e28be3-93be-4b97-9373-23c18e6fe8e6 (p2, c1, k15).
2022-09-16T11:37:25.855225600Z 1663328245: New connection from 172.18.0.26 on port 1883.
2022-09-16T11:37:25.911007600Z 1663328245: New client connected from 172.18.0.26 as DataExporter-42a8439d-2c47-4530-957e-ddbdb79b720a (p2, c1, k15).
2022-09-21T10:57:05.559838000Z 1663757825: Socket error on client HartIP-44e28be3-93be-4b97-9373-23c18e6fe8e6, disconnecting.
2022-09-21T10:58:22.590406800Z 1663757902: New connection from 172.18.0.26 on port 1883.
2022-09-21T10:58:22.593113700Z 1663757902: New client connected from 172.18.0.26 as HartIP-44e28be3-93be-4b97-9373-23c18e6fe8e6 (p2, c1, k15).
2022-09-21T14:36:25.491329500Z 1663770985: Socket error on client HartIP-44e28be3-93be-4b97-9373-23c18e6fe8e6, disconnecting.
2022-09-21T14:36:36.500793400Z 1663770996: New connection from 172.18.0.26 on port 1883.
2022-09-21T14:36:37.498801900Z 1663770997: New client connected from 172.18.0.26 as HartIP-44e28be3-93be-4b97-9373-23c18e6fe8e6 (p2, c1, k15).
2022-09-21T15:04:31.036505700Z 1663772671: Socket error on client HartIP-44e28be3-93be-4b97-9373-23c18e6fe8e6, disconnecting.

The error log keeps repeating until I got this:

2022-09-22T03:21:10.288284900Z 1663816870: Socket error on client <unknown>, disconnecting.
2022-09-22T03:21:22.826682100Z 1663816882: New connection from 172.18.0.26 on port 1883.

The mosquitto container has a config file like below:

log_type error
log_type warning
log_type notice
log_type information
#log_type all
#log_type debug

log_timestamp true

#log_dest file /mosquitto/log/mosquitto.log
#user root
log_dest stdout
sys_interval 300

Based on my researches on other issues, I have tried using netstat & lsof:

developer@localhost:~$ sudo netstat -tanp | grep :1883
tcp        0      0 0.0.0.0:1883            0.0.0.0:*               LISTEN      5811/docker-proxy
tcp        0      0 127.0.0.1:41980         127.0.0.1:1883          ESTABLISHED 13919/./amd64
tcp        0      0 127.0.0.1:1883          127.0.0.1:41980         ESTABLISHED 5811/docker-proxy
tcp        0      0 172.18.0.1:43212        172.18.0.3:1883         ESTABLISHED 5811/docker-proxy
tcp6       0      0 :::1883                 :::*                    LISTEN      5818/docker-proxy
developer@localhost:~$ sudo lsof -i:1883
COMMAND     PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
docker-pr  5811 root    3u  IPv4 77828754      0t0  TCP localhost:1883->localhost:41980 (ESTABLISHED)
docker-pr  5811 root    4u  IPv4    50201      0t0  TCP *:1883 (LISTEN)
docker-pr  5811 root    5u  IPv4 77828756      0t0  TCP 172.18.0.1:43212->172.18.0.3:1883 (ESTABLISHED)
docker-pr  5818 root    4u  IPv6    50205      0t0  TCP *:1883 (LISTEN)
amd64  13919 root    3u  IPv4 77827400      0t0  TCP localhost:41980->localhost:1883 (ESTABLISHED)

I do not have control over the mosquitto container as it is bundled in the Ubuntu host given by client. However, I could not find anywhere else to troubleshoot this random bug.

1. What else can I do to get more information about this bug?
2. Is the problem due to mosquitto container or is it from MQTTnet.Extensions.ManagedClient taken from this repo

I am happy to send over more log & info upon requests.