[WARN]: tcp: tls: first record does not look like a TLS handshake when terminating TLS

[sbrl]

When I configure Fabio to terminate TLS for a Mosquitto MQTT server, I get the following error:

Dec 14 16:55:59 SERVER_NAME fabio[29480]: 2021/12/14 16:55:59 [WARN]: tcp:  tls: first record does not look like a TLS handshake

My setup is such:

• Fabio runs a the reverse proxy, and accepts MQTTS connections on port 8883 and terminates TLS.
• Fabio then makes unencrypted MQTT requests to the backend MQTT server listening on port 1883 on behalf of the client.
• Consul is obviously in use to keep track of services.
• Nomad + Docker are being used to run services.

The following instructions were derived from the official documentation at https://fabiolb.net/feature/tcp-proxy/.

Steps to reproduce

1. Setup a Mosquitto MQTT server (sudo apt install mosquitto, more in depth tutorial, part 2)
2. Register 2 services in Consul. mqtts with the tags urlprefix-:8883 and proto=tcp, and mqtt with the tags urlprefix-:1883 and proto=tcp
3. Configure Fabio with this:

proxy.addr = :1883;proto=tcp,:8883;proto=tcp;cs=your_valid_certstore;

4. Start Fabio and the mosquitto MQTT server
5. Attempt to connect with mosquitto_sub (sudo apt install mosquitto-clients):

mosquitto_sub -h mqtt.yourdomain.com -p 8883 -t '#' -v
# Or, with authentication setup:
mosquitto_sub -h mqtt.yourdomain.com -p 8883 -t '#' -v -u youruser -P yourpassword

6. Notice how the logs of Fabio contain many rows of this:

Dec 14 17:00:22 SERVER_NAME fabio[29578]: 2021/12/14 17:00:22 [WARN]: tcp:  tls: first record does not look like a TLS handshake

7. Notice how the mosquitto logs look like this:

1639501352: New connection from FABIO_IP_ADDR on port 1883.
1639501352: Socket error on client <unknown>, disconnecting.

8. Now attempt to connect with unencrypted MQTT:

mosquitto_sub -h mqtt.yourdomain.com -p 1883 -t '#' -v
# Or, with authentication setup:
mosquitto_sub -h mqtt.yourdomain.com -p 1883 -t '#' -v -u youruser -P yourpassword

9. Notice how it connects successfully. Try publishing a message:

mosquitto_pub -h mqtt.yourdomain.com -p 8883 -t 'some/topic' -m "a test message" -v
# Or, with authentication setup:
mosquitto_pub -h mqtt.yourdomain.com -p 8883 -t 'some/topic' -m "a test message" -v -u youruser -P yourpassword

Additional system info:

• uname -a: Linux SERVER_NAME 5.10.52-v7l+ #1441 SMP Tue Aug 3 18:11:56 BST 2021 armv7l GNU/Linux
• fabio -v: 1.5.15
• Mosquitto version: 1.5.7

[nathanejohnson]

Your fabio configuration looks correct, but the fabio message is telling you that your mosquito client is not trying to do a TLS handshake. I am not familiar enough with mosquito to troubleshoot this issue.

[nathanejohnson]

Okay, looking at the mosquito docs, 8883 is a "magic" port that denotes tcps. Fabio will match the 8883 service for mqtt, which expects encryption. So fabio is terminating TLS, then sending to the encrypted mosquito port of 8883. One option would be to set up only one service entry for consul, do urlprefix tag be urlprefix-:8883, and it could also have the tag of urlprefix-:1883, and have the service port actually be 1883. Don't register a consul service for port 8883. This will have the effect of forwarding traffic that came from 8883 to be forwarded to the service port 1883 after terminating TLS, and forwarding 1883 as-is (no TLS). Anyway, that's about as far as I can go with my current knowledge of moqsuito

[sbrl]

Hmmm, that makes things worse @nathanejohnson. I tried having a single tag urlprefix-:8883 urlprefix-:1883 proto=tcp, and also 2 tags urlprefix-:8883 proto=tcp and urlprefix-:1883 proto=tcp on a single service, but it claims in the routing table:

...whereas with 2 separate service definitions I get this:

For reference, the service definitions in my mqtt.nomad file:

service {
	name = "mqtt"
	task = "mqtt"
	tags = [
		"service",
		"urlprefix-:1883 proto=tcp", // mqtt.mooncarrot.space
	]
	address_mode = "host"
	port = "mqtt"
	
	check {
		type = "tcp"
		port = "mqtt"
		interval = "60s"
		timeout = "10s"
	}
}
service {
	name = "mqtts"
	task = "mqtt"
	tags = [
		"service",
		"urlprefix-:8883 proto=tcp" // mqtt.mooncarrot.space; TLS termination is configured in fabio.properties
	]
	address_mode = "host"
	port = "mqtt"
	
	check {
		type = "tcp"
		port = "mqtt"
		interval = "60s"
		timeout = "10s"
	}
}

[nathanejohnson]

That's... really odd. Two urlprefix inside the same tag will never work, but I'm not sure why two urlprefix tags on the same service isn't. With that said, does having two services on port 1833 but with different urlprefix tags actually work as expected?