mosquitto can't connect when TLS is enabled

[tehXor]

Describe the bug
When enabling TLS (with the use of Let's Encrypt certificates) mosquitto is unable to connect. After Client (null) sending CONNECT it just throws an Error: Unknown error. without further output, although the -d flag for debug output is set. Connecting with TLS disabled on the MQTT adapter works. Connecting with openssl s_client -connect and TLS enabled on the MQTT adapter also looks good and every certificate in the chain could be verified. Even with the debug flag the MQTT adapter log shows no entries of these failed connections.

To Reproduce
Steps to reproduce the behavior:

1. In the adapter settings tick the SSL checkbox and use public, private and chain certificate files from Let's Encrypt for your domain under which ioBroker is reachable (and optionally set the port to 8883 to ensure clients will pick up that they should use TLS).
2. On another machine install all mosquitto packages: apt install mosquitto*
3. Try to connect with a test message (and optionally TLS flags to ensure mosquitto tries to use TLS): mosquitto_pub -h my.iobroker.domain -t test -m "hello" -p 8883 -u "test" -P "t35t" --tls-use-os-certs -d (you can also try all other TLS related flags)
4. This always results in an Error: Unknown error. no matter which mosquitto flags you try.

Expected behavior
It should publish the test message just like it does when TLS (the SSL flag in the MQTT adapter options) is disabled.

Screenshots & Logfiles
mosquitto_pub -h my.iobroker.domain -t test -m "hello" -p 8883 -u "test" -P "t35t" --tls-use-os-certs -d:

Client (null) sending CONNECT
Error: Unknown error.

openssl s_client -connect my.iobroker.domain:8883:

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.iobroker.domain
verify return:1
---
Certificate chain
 0 s:CN = *.iobroker.domain
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
XXXXXXXXXXXXXXXXXXXXXXXXXXX
-----END CERTIFICATE-----
subject=CN = *.iobroker.domain

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5088 bytes and written 385 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: XXXXXXXXXXXXXXX
    Session-ID-ctx:
    Resumption PSK: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX   XXXXXXXXXXXXXXXX

    Start Time: 1670990241
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: XXXXXXXXXXXXXXX
    Session-ID-ctx:
    Resumption PSK: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX   XXXXXXXXXXXXXXXX

    Start Time: 1670990241
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

Adapter log:

mqtt.0 | 2022-12-14 05:27:16.479 | info | Starting MQTT (Secure) authenticated server on 0.0.0.0:8883
-- | -- | -- | --
mqtt.0 | 2022-12-14 05:27:16.297 | info | starting. Version 4.0.7 in /opt/iobroker/node_modules/iobroker.mqtt, node: v16.18.1, js-controller: 4.0.23
mqtt.0 | 2022-12-14 05:27:16.236 | info | Plugin sentry Sentry Plugin disabled for this process because sending of statistic data is disabled for the system

Versions:

• Adapter version: 4.0.7
• Operating system: Ubuntu, Debian Bullseye

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within the next 7 days. Please check if the issue is still relevant in the most current version of the adapter and tell us. Also check that all relevant details, logs and reproduction steps are included and update them if needed. Thank you for your contributions. Dieses Problem wurde automatisch als veraltet markiert, da es in letzter Zeit keine Aktivitäten gab. Es wird geschlossen, wenn nicht innerhalb der nächsten 7 Tage weitere Aktivitäten stattfinden. Bitte überprüft, ob das Problem auch in der aktuellsten Version des Adapters noch relevant ist, und teilt uns dies mit. Überprüft auch, ob alle relevanten Details, Logs und Reproduktionsschritte enthalten sind bzw. aktualisiert diese. Vielen Dank für Eure Unterstützung.

[bitboy00]

same here ... TLS/SSL doesn't work if used as Server.

[mcm1957]

@Apollon77 Anybody maintaining this adapter?

[Apollon77]

I think first of all ... a debug level log would be awesome because I do not see any info here about this unknown error anywhere ... Also is that error thrown on iobroker side or on mosquitto side?

[bitboy00]

A debug (silly) level log shows exactly nothing. It just doesn't work. There is a tcpdump showing that there is a partial handshake but the client (whatever the client is) disconnects for an unknown reason. That might be mqtt-explorer or any other client.

The most basic test:

Create a server mqtt.0 and a client mqtt.1 on the same server. Connect using 127.0.0.1 without TLS and the client and the Server immediately are “green”. Data is running just fine.

Configure both sides SSL(TLS) and try to connect again. The client shows “green” but the server is “orange” ... nothing will happen, and the log shows nothing interesting.

I tried Letsencrypt-Certs and self-signed ones with different Parameters. Now, I installed mosquitto, and it runs just fine with that very certificates. But there is no Admin-integration, of course. Any Client is okay with that. The mqtt.adapter as client like the MQTT-explorer.

[Apollon77]

Then it is about using wireshark or such and compare between a working and a non working solution to maybe find out whats the difference or find out how to get more detailed errors from mosquitto when they report the "Unknown error", or?!

[bitboy00]

In my opinion, that's way more than a developer can wait for. I'm user and spent hours with testing and logging and creating certs. An as even a local connect using address 127.0.0.1 is not working... And mosquito is functioning. There are no visible errors. Nothing in the Logofiles.

So if you have interest in providing a working Adapter, you can reproduce it and will find bugs faster than I could do. Otherwise my

[Apollon77]

So if you have interest in providing a working Adapter ...

Sure we have ... always ... for anyone of the 200+ core managed adapters ... That's why the issue is open. Somewhen someone will have the time to look into it ( just to be sure we see it the same ... 2 users ot of 27k installations of this adapter have this issue rn ..... right?

The more help and info we can get the faster it can be to find and understand the issue.

[bitboy00]

I can't help you with that. It's 5 minutes to reproduce, and it will tell you much faster than me where to look for. I tried on a raspberry pi 3 an 4 and also a VM with 4 cores, 32gb and SSD. Raspian and Debian bookworm. I followed the available manuals, YouTube tutorials and ask Google and chatgpt. It just can't connect if I switched on SSL. Never. And there where no visible errors in a log.

Meanwhile I use mosquito, even if it's not integrated with admin. Your mqtt.adapter-client connects just fine to it using SSL/TLS. It just worked from the first try.

So don't blame me. I'm out here.