libressl
With SSL renegotiation, openssl CLI fails with "called a function you should not call"
When using libressl with the command openssl on the command line with -starttls and during the session, a renegotiation occurs, the openssl command line tool aborts with the following error message:
40619255824:error:1404C042:SSL routines:ST_OK:called a function you should not call:/usr/src/lib/libssl/ssl_lib.c:2862:
This should not happen.
Issue reproduced on OpenBSD 7.7 with LibreSSL 4.1.0 (default install) on both ARM64 and x86_64 Issue reproduced on OpenBSD 7.6 with LibreSSL 4.0.0 (default install) on both ARM64 and x86_64 Issue reproduced on OpenBSD 7.6 with LibreSSL 4.0.0 (default install) on x86_64
There are two ways to reproduce.
Variant 1:
$ openssl s_client -starttls ftp -crlf -connect test.rebex.net:21
CONNECTED(00000003)
(... some stuff removed ...)
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID:
Session-ID-ctx:
Master-Key:
Start Time: 1746077388
Timeout : 7200 (sec)
Verify return code: 0 (ok)
---
See https://test.rebex.net/ for more information and terms of use.
USER ftp
331 Anonymous login OK, send your complete email address as your password.
PASS [email protected]
230 User 'ftp' logged in.
REIN
RENEGOTIATING
17370740397880:error:1404C042:SSL routines:ST_OK:called a function you should not call:/usr/src/lib/libssl/ssl_lib.c:2862:
Variant 2:
Configure SMTP-Server, e.g. OpenSMTPd or any other (tested against several systems), with either self-signed or CA-signed key/cert (irrelevant to trigger the issue), e.g.,
pki openbsd77 key "/etc/ssl/private/testmailkey.pem"
pki openbsd77 cert "/etc/ssl/testmailcert.pem"
listen on lo0 inet4 port 25 hostname openbsd77 tls pki openbsd77
and do:
$ openssl s_client -starttls smtp -crlf -connect 127.0.0.1:25
CONNECTED(00000003)
(... some stuff removed ...)
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_CHACHA20_POLY1305_SHA256
Session-ID:
Session-ID-ctx:
Master-Key:
Start Time: 1746076742
Timeout : 7200 (sec)
Verify return code: 18 (self signed certificate)
---
250 HELP
EHLO Test
250-openbsd77 Hello Test [127.0.0.1], pleased to meet you
250-8BITMIME
250-ENHANCEDSTATUSCODES
250-SIZE 36700160
250-DSN
250 HELP
MAIL FROM: <[email protected]>
250 2.0.0 Ok
RCPT TO: <openbsd77@localhost>
RENEGOTIATING
95429802000:error:1404C042:SSL routines:ST_OK:called a function you should not call:/usr/src/lib/libssl/ssl_lib.c:2862:
Thanks. I think this is mostly expected since the generic TLS methods set .ssl_renegotiate to ssl_undefined_function(). If you pass -tls1_2 to force a method supporting renegotiation this works.
We could intercept attempts to renegotiate with TLSv1.3 (which doesn't support renegotiation) either in SSL_renegotiate*() in ssl_lib.c before this handler is called or in the openssl command line tool, or both, to throw a friendlier error like OpenSSL 3 does.
A friendlier error (ideally pointing to -tls1_2) would be a good idea, however, this does not solve the issue:
$ openssl s_client -starttls smtp -crlf -connect 127.0.0.1:25 -legacy_renegotiation -tls1_2
CONNECTED(00000003)
(... some stuff removed ...)
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-CHACHA20-POLY1305
Session-ID:
Session-ID-ctx:
Master-Key: 92C9C1584E718B53338BF2DB3AF19F6A5ADA89929F225E47A1F12314C22ED236C476CC905F64A66B539DA3A4D89B57B5
Start Time: 1746097053
Timeout : 7200 (sec)
Verify return code: 18 (self signed certificate)
---
250 HELP
EHLO test
250-openbsd77 Hello test [127.0.0.1], pleased to meet you
250-8BITMIME
250-ENHANCEDSTATUSCODES
250-SIZE 36700160
250-DSN
250 HELP
MAIL FROM: <[email protected]>
250 2.0.0 Ok
RCPT TO: <user@localhost>
RENEGOTIATING
147267745808:error:1400444C:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert no renegotiation:/usr/src/lib/libssl/ssl_pkt.c:753:SSL alert number 100
147267745808:error:140040E5:SSL routines:CONNECT_CR_SRVR_HELLO:ssl handshake failure:/usr/src/lib/libssl/ssl_pkt.c:495:
The error occurs with and without -legacy_renegotiation
It is a bit confusing, since OpenSMTPd runs on the same machine and therefore uses the same LibreSSL. So why would it at one end of the connection request a renegotiation, if it does not support it on the other end?
$ ldd /usr/sbin/smtpd
/usr/sbin/smtpd:
Start End Type Open Ref GrpRef Name
0000001cffdd0000 0000001cffe85000 exe 1 0 0 /usr/sbin/smtpd
000000217df0c000 000000217df4a000 rlib 0 1 0 /usr/lib/libevent.so.4.1
000000216b312000 000000216b358000 rlib 0 1 0 /usr/lib/libutil.so.20.0
0000002102c3d000 0000002102c81000 rlib 0 1 0 /usr/lib/libtls.so.32.1
0000002121620000 00000021216b9000 rlib 0 2 0 /usr/lib/libssl.so.59.1
00000021858a9000 0000002185b00000 rlib 0 3 0 /usr/lib/libcrypto.so.56.0
00000021b7739000 00000021b7785000 rlib 0 1 0 /usr/lib/libz.so.7.1
0000002103839000 0000002103964000 rlib 0 1 0 /usr/lib/libc.so.100.3
00000021c23b0000 00000021c23b0000 ld.so 0 1 0 /usr/libexec/ld.so
# ldd `whereis openssl`
/usr/bin/openssl:
Start End Type Open Ref GrpRef Name
0000000230ca0000 0000000230d60000 exe 1 0 0 /usr/bin/openssl
00000006d79df000 00000006d7a78000 rlib 0 1 0 /usr/lib/libssl.so.59.1
0000000674dbd000 0000000675014000 rlib 0 2 0 /usr/lib/libcrypto.so.56.0
000000072c708000 000000072c833000 rlib 0 1 0 /usr/lib/libc.so.100.3
00000006f3f10000 00000006f3f10000 ld.so 0 1 0 /usr/libexec/ld.so
It's the lovely s_client behavior that hurts you now:
When used interactively (which means neither -quiet nor -ign_eof have been given), the session will be renegotiated if the line begins with an R; if the line begins with a Q or if end of file is reached, the connection will be closed down.
So your line RCPT TO: ... eats the first R and requests a renegotiation instead of sending 'RCPT TO' to the server. The server denies that request with an no_renegotiation(100) alert since OpenSMTPd uses libtls, which sets SSL_OP_NO_CLIENT_RENEGOTIATION on servers because that's a classic attack vector (look up 'renegotiation attack').
You're probably better off experimenting with a nc(1)-like tool (although inserting the CRLF line endings can be slightly annoying with that).
Well, should s_client really listen to the content? Since any "R" as first letter of a line would then initiate a renegotiation. Which it doesn't on openssl.
Well, it's the historical behavior and OpenSSL changed that only rather recently as far as I can tell (PR 20566 from two years ago, unless I'm mistaken).
Whether it's a feature worth spending time on, I don't know. I would not be against landing a patch that matches this behavior, but I doubt we will be doing this ourselves.