Authoritative server 4.7.4 on RHEL8 can't use SSL with mariadb

[GuyT2002]

• Program: Authoritative version 4.7.4, using mysql-backend (mariadb 10.11.4) on RHEL8
• Issue type: Bug report

Short description

Trying to start an authoritative 4.7.4 server on RHEL8, with gmysql-backend (mariadb 10.11.4, also on RHEL8). However, every time I try to start the service, I recieve the following issue:

gmysql Connection failed: Unable to connect to database: ERROR 1043 (08S01): Bad handshake
Caught an exception instantiating a backend: Unable to launch gmysql connection: Unable to connect to database: ERROR 1043 (08S01): Bad handshake

Environment

• Operating system: RHEL8
• Software version: PDNS authoritative 4.7.4, mariadb 10.11.4
• Software source: PDNS repohttps://repo.powerdns.com/el/$basearch/$releasever/auth-47/debug Mariadb repo: https://mirror.yer.az/mariadb/yum/10.11/rhel/$releasever/$basearch

PDNS.conf file:

launch=gmysql

gmysql-host=192.168.121.162
gmysql-user=pdnsuser
gmysql-dbname=pdns
gmysql-password=pdnsuser123
gmysql-ssl=yes
api=yes
api-key=76c76bcc2e870735c
webserver-address=0.0.0.0
webserver-allow-from=192.168.0.0/16,127.0.0.1,10.0.0.0/8
security-poll-suffix=
loglevel=5
log-dns-queries=yes

#resolver=[::1]:53
expand-alias=yes
default-soa-content=a.misconfigured.dns.server.invalid hostmaster.@ 0 10800 3600 604800 300

mariadb has a database called pdns with all necessary tables (created via the installation guide). it is important to note, I can use mysql to access the mariadb server from the PDNS auth machine just fine:

[vagrant@pdns ~]$ 
[vagrant@pdns ~]$ mysql -h 192.168.121.162 -updnsuser -p
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 21
Server version: 10.11.4-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> 
MariaDB [(none)]>

Steps to reproduce

1. create 2 RHEL8 nodes, one with mariadb 10.11.4, one with PDNS authoritative server 4.7.4
2. Try to use the mariadb as a backend for the PDNS server (With SSL!) and receive the same error when trying to run the pdns server.

Expected behaviour

PDNS server running without any issue.

Actual behaviour

receiving this SSL handshake error upon startup, and failing.

Other information

I've already discussed with the IRC support, but I could not find any workaround:

1. using update-crypto-policies does not solve the issue.
2. Trying to install a newer version of mariadb-connector-c is impossible, since the latest version which is support in EL8 is mariadb-connector-c.x86_64 3.1.11-2.el8_3.

[Habbie]

it is important to note, I can use mysql to access the mariadb server from the PDNS auth machine just fine:

Does this connection use SSL?

gmysql Connection failed: Unable to connect to database: ERROR 1043 (08S01): Bad handshake

Can you get a pcap of this handshake? (using a tool such as tcpdump)

[GuyT2002]

Does this connection use SSL?

yes

Can you get a pcap of this handshake? (using a tool such as tcpdump)

Sure, adding here

output.pcap.txt

[Habbie]

Thanks. It looks like mariadb is -opening- the SSL part of the connection with a bad handshake error. Can you compare this handshake to one from the mysql -h ... exchange?

[GuyT2002]

output-cmd.pcap.txt This is it

[rgacogne]

It looks like the connection initiated by pdns w/ libmysql-client doesn't actually start the TLS handshake, and the server gives up after 10s, sending the "Bad handshake" message.

[Habbie]

ah yes! I forgot to look at the timestamps

[Habbie]

so it looks like pdns isn't using SSL at all then.. although, I don't know how the SSL upgrade is supposed to work.

[rgacogne]

It looks like we pass CLIENT_SSL to mylsq_real_connect()'s flags, but I see that the MySQL documentation tells us not to do that:

CLIENT_SSL: Use SSL (encrypted protocol). Do not set this option within an application program; it is set internally in the client library. Instead, use mysql_options() or mysql_ssl_set() before calling mysql_real_connect().

And mariadb doesn't list the flag at all: https://mariadb.com/kb/en/mysql_real_connect/

So perhaps this is no longer supported?

[Habbie]

Trying to install a newer version of mariadb-connector-c is impossible, since the latest version which is support in EL8 is mariadb-connector-c.x86_64 3.1.11-2.el8_3.

is the mariadb cmd client using this same lib?

(As I typed that question, @rgacogne's comment appeared, so perhaps the question is not that useful anymore)

So perhaps this is no longer supported?

this seems likely

[GuyT2002]

is the mariadb cmd client using this same lib?

Are you asking me?, All their libraries are provided via the following packages:

MariaDB-client.x86_64
MariaDB-common.x86_64
MariaDB-shared.x86_64
pdns-backend-mysql.x86_64

Now as far as I can tell, mariadb-connector-c is provided via this package: MariaDB-shared-10.11.4-1.el8.x86_64 But the CMD client (the mysql command) is provided from this package: MariaDB-client-10.11.4-1.el8.x86_64 This are supposed to work with one another as far as mariadb explain in their website.

From you've both wrote, you suspect that mariadb libraries themselves stopped supporting SSL with the mairadb-connector-c lib?

[Habbie]

From you've both wrote, you suspect that mariadb libraries themselves stopped supporting SSL with the mairadb-connector-c lib?

We're suspecting that the way -we- enable SSL is no longer supported.

[Habbie]

Can you share your mariadb server config?

[Habbie]

I swear this is not a joke: can you try with gmysql-ssl=no ?

[Habbie]

#13060 might fix this. It does so on my system anyway :)

[Habbie]

(once you have a working connection, you can use pdnsutil backend-cmd gmysql 'show session status like "ssl_cipher"' to see if pdns is using SSL towards your database)

[Habbie]

I just noticed this is somewhat of a duplicate of #12528

[dg410]

Hi, tried to use gmysql=no on a different, disconnected environment with the same setup. The service didn't fail, but is not using ssl. I used tcpdump and saw that the connection is unencrypted, and pdnsutil backend-cmd returned no ssl cipher.

Our mariadb configuration is very basic, we only configured ca cert, server cert and key and client cert and key. I'll be able to share it on Sunday. How can we use 13060? Is there a build or something like that?

btw we didn't know pdnsutil backend-cmd gmysql exists and that's so cool.

[Habbie]

btw we didn't know pdnsutil backend-cmd gmysql exists and that's so cool.

I also keep forgetting about it! (.. and I wrote it :D )

The service didn't fail, but is not using ssl

can you try:

• setting gmysql-group=pdns in `pdns.conf
• adding a [pdns] group to my.cnf
• putting at least ssl, plus anything else that seems necessary from https://mariadb.com/kb/en/securing-connections-for-client-and-server/#enabling-tls-for-mariadb-clients in that group?

(with your feedback, I'll be able to update our documentation to allow people to use SSL again)

How can we use 13060? Is there a build or something like that?

Not currently, it doesn't even build for some platforms. I might ask you to test a patch later, depending on how the my.cnf group experiment above goes.

[dg410]

Hi, configured pdns.conf

launch=gmysql

gmysql-host=<host>
gmysql-user=<user>
gmysql-dbname=<db>
gmysql-password=<pass>
gmysql-ssl=no
gmysql-group=pdns
api=yes
api-key=<key>
webserver-address=0.0.0.0
webserver-allow-from=<ip>
security-poll-suffix=
loglevel=7
log-dns-queries=yes

#resolver=[::1]:53
expand-alias=yes
default-soa-content=a.misconfigured.dns.server.invalid hostmaster.@ 0 10800 3600 604800 300

and /etc/my.cnf

#
# This group is read both by the client and the server
# use it for options that affect everything
#
[mariadb]
ssl-cert = /etc/ssl/mysql/server-cert.pem
ssl-key = /etc/ssl/mysql/server-key.pem
ssl-ca = /etc/ssl/mysql/ca-cert.pem

[client]
ssl-cert = /etc/ssl/mysql/client-cert.pem
ssl-key = /etc/ssl/mysql/client-key.pem
ssl-ca = /etc/ssl/mysql/ca-cert.pem

[pdns]
ssl-cert = /etc/ssl/mysql/client-cert.pem
ssl-key = /etc/ssl/mysql/client-key.pem
ssl-ca = /etc/ssl/mysql/ca-cert.pem

With gmysql-ssl=no gmysql-group=pdns:

dnsutil backend-cmd gmysql 'show session status like "ssl_cipher"'
Jul 23 11:50:34 gmysql Connection successful. Connected to database '<db>' on '<host>'.
Jul 23 11:50:34 gmysql Connection successful. Connected to database '<db>' on '<host>'.
== show session status like "ssl_cipher"
'Ssl_cipher'	''	

With gmysql-ssl=yes gmysql-group=pdns the service fails.

Do we need to configure something else under the pdns group?

[Habbie]

can you try just adding a line ssl to the pdns group?

[dg410]

[mariadb]
ssl-cert = /etc/ssl/mysql/server-cert.pem
ssl-key = /etc/ssl/mysql/server-key.pem
ssl-ca = /etc/ssl/mysql/ca-cert.pem

[client]
ssl-cert = /etc/ssl/mysql/client-cert.pem
ssl-key = /etc/ssl/mysql/client-key.pem
ssl-ca = /etc/ssl/mysql/ca-cert.pem

[pdns]
ssl

Like this? pdns still doesn't use ssl-

== show session status like "ssl_cipher"
'Ssl_cipher'	''	

[Habbie]

Thank you for testing. I'll dig some more.

[Habbie]

Just to be sure - this is my.cnf on the client (pdns) machine, and pdns can read it?

[dg410]

No, we configured my.cnf only on the DB machine, which works with rhel7. Configured my.cnf on the pdns machine as well:

1. gmysql-ssl=yes, no group, empty /etc/my.cnf- bad handshake, service failed

2. gmysql-ssl=no, no group, empty /etc/my.cnf- no ssl

3. gmysql-ssl=no, gmysql-group=pdns, empty /etc/my.cnf- no ssl

4. gmysql-ssl=yes, gmysql-group=pdns, empty /etc/my.cnf- bad handshake, service failed

5. gmysql-ssl=yes, no group, ssl configured in /etc/my.cnf under [client]- bad handshake, service failed

6. gmysql-ssl=no, no group, ssl configured in /etc/my.cnf under [client]- no ssl

7. gmysql-ssl=yes, no group, ssl-cert, ssl-key and ssl-ca configured in /etc/my.cnf under [client]- works with ssl

8. gmysql-ssl=no, no group, ssl-cert, ssl-key and ssl-ca configured in /etc/my.cnf under [client]- works with ssl

9. gmysql-ssl=yes, gmysql-group=pdns, ssl-cert, ssl-key and ssl-ca configured in /etc/my.cnf- works with ssl

10. gmysql-ssl=no, gmysql-group=pdns, ssl-cert, ssl-key and ssl-ca configured in /etc/my.cnf- works with ssl

So looks like gmysql-group doesn't matter and we need the client key, certificate and ca for it to work. We'll run some more tests, do you think adding the certs will mean mutual tls or just server tls?

[Habbie]

Glad to hear you found some working combinations! Also good to hear that gmysql-ssl=yes appears to be harmless in those cases.

gmysql-group=pdns should tell the client library to load the pdns section instead of the client section - does that help any misunderstanding?

[Habbie]

We'll run some more tests, do you think adding the certs will mean mutual tls or just server tls?

I don't know, but https://mariadb.com/kb/en/securing-connections-for-client-and-server/#enabling-tls-for-mariadb-clients which I linked earlier has a section "Enabling Two-Way TLS for MariaDB Clients"

[mstenz]

I have just installed PDNS and I am forced to use an old version of MySQL (5.1).

I also get the same error:

Caught an exception instantiating a backend: Unable to launch gmysql connection: Unable to connect to database: ERROR 1043 (08S01): Bad handshake

I don't want to use ssl and have disabled this in the mysql completely, but the error does not go away.

i do not have a /etc/my.cnf file (Ubuntu 22.04 LTS)

Adding gmysql-ssl=no to the pdns config doesnt seem to disable ssl for the client?

What I can do? What mysqlclient version is PDNS using?

[dmachard]

Hi @Habbie

We have exactly the same issue regarding gmysql-ssl=yes but after some tests, I successed to enable TLS :) I can share some details, I did the test with the docker image powerdns/pdns-auth-48:4.8.2

Below my pdns gmysql backend config

launch=gmysql
gmysql-host=172.16.0.10
gmysql-dbname=pdns
gmysql-user=pdns_user
gmysql-password=pdns_secret
gmysql-group=pdns

I added the /etc/my.cnf file. In my case, client key, certificate are mandatory.

[pdns]
ssl-cert = /etc/powerdns/client-cert.pem
ssl-key = /etc/powerdns/client-key.pem

with that, the connection is successful and properly enforced with TLS.

$ sudo docker exec docker-stack-dns-ns-1 pdnsutil backend-cmd gmysql 'show session status like "ssl_cipher"'
== show session status like "ssl_cipher"
'Ssl_cipher'	'TLS_AES_256_GCM_SHA384'

I can make more tests if you need

[Habbie]

Thanks Denis, between your comment and @dg410's tests earlier I have enough to remove the flag and fix the docs!

[Habbie]

https://dmachard.github.io/posts/0058-pdns-tls-mysql/ :heart: