properly report expired CRLs when verifying client certificates

[sbiberhofer]

Output of haproxy -vv and uname -a

HA-Proxy version 1.9.6 2019/03/29 - https://haproxy.org/
Build options :
  TARGET  = freebsd
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.1a-freebsd  20 Nov 2018
Running on OpenSSL version : OpenSSL 1.1.1a-freebsd  20 Nov 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.43 2019-02-23
Running on PCRE version : 8.43 2019-02-23
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with multi-threading support.

Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE
              h2 : mode=HTX        side=FE|BE
       <default> : mode=HTX        side=FE|BE
       <default> : mode=TCP|HTTP   side=FE|BE

Available filters :
        [SPOE] spoe
        [COMP] compression
        [CACHE] cache
        [TRACE] trace

FreeBSD skyforge.at 12.0-RELEASE-p3 FreeBSD 12.0-RELEASE-p3 GENERIC  amd64  

What's the configuration?

The essential part is the client cert verification in the frontend, e.g.:

frontend submission
        bind *:465 ssl crt server.pem crl-file crl.pem ca-file ca.crt verify required
        bind :::465 ssl crt server.pem crl-file crl.pem ca-file ca.crt verify required
        mode tcp
        default_backend submission

Steps to reproduce the behavior

1. Configure a haproxy frontend w/ tls and mandatory client certificate verification
2. Configure a CRL file for haproxy to detect revoked certificates
3. Wait until the CRL file expires
4. Have a client connect to haproxy and present its client certificate
5. haproxy will report the certificate as expired to the client and log it as untrusted instead of reporting errors w/ the CRL file.

Actual behavior

haproxy reports a certificate expired (45) error to the client. At the same time it usually logs a "SSL cient certificate not trusted" line to the haproxy log. Running haproxy w/ -d for debugging yields

fd[0009] OpenSSL error[0x14094412] ssl3_read_bytes: sslv3 alert bad certificate
fd[0009] OpenSSL error[0x1417c086] tls_process_client_certificate: certificate verify failed
fd[0009] OpenSSL error[0x1417c086] tls_process_client_certificate: certificate verify failed
fd[0009] OpenSSL error[0x1417c086] tls_process_client_certificate: certificate verify failed

All of this seemingly points to problems in the client's certificate instead of the expired CRL.

Expected behavior

I would prefer haproxy to properly log a message indicating a problem w/ the CRL file instead of logging a rather generic error message which seemingly shifts the blame to the client.

This seems even more important as quick ways to verify client certificates (such as openssl verify when used without the -crl_check_all extra parameter) don't report expired CRLs and instead mark the vertificate as "valid". Even worse, other servers, such as gnutls-serv ignore expired CRLs and accept these client certs. This made it extremely hard to find the cause for the problem, as haproxy was quite singular (albeit probably correct) in it's behaviour.

Do you have any idea what may have caused this?

Do you have an idea how to solve the issue?

haproxy should either keep track of the CRL lifetime and issue warnings after expiry, or properly check CRL validity when evaluating client certificates. In any case, a proper warning message would be awesome.

[lukastribus]

Tagging this as a feature request, since this is a change from the current designed behavior.

From a end-user perspective I agree with you, the SSL errors need to be more accessible (can't always run full debug mode in production) and they are also ambiguous.

The problem is that this is not within our control. We are using OpenSSL to handle everything SSL related and those error messages are directly triggered by OpenSSL, see sslv3 alert bad certificate and certificate verify failed.

Haproxy is just looping through the error stack and printing them.

We would have to:

• actually log SSL errors on some lower log level, not only in the actual debug mode (this should be doable and would simplify SSL troubleshooting a lot, but needs discussion)
• but more importantly, the OpenSSL library has to actually provide useful error codes and strings, otherwise you just see those ambiguous sslv3 alert bad certificate and tls_process_client_certificate: certificate verify failed on a different log level

The former is something that needs to be discussed here or on the mailing list. The latter is something that needs to be done at OpenSSL. And that may be a long shot. You can try filing an issue with them, explaining the situation and showing the debug output from above.

[mshahat]

@lukastribus I guess I agree mostly with your comment. I found debugging SSL issues on haproxy extremely difficult, possibly as a result of the reasons you've explained. Thanks,

[wlallemand]

Where it may be complicated to report a precise error with the OpenSSL library for each request, it is probably doable to have something on the CLI which allows to see the status of the CRL.

[huntc]

As an adjunct to this issue, I had an SSL cert expire on a backend that had been configured as ssl verify none. Yet, when that backend's cert expired, the expiry propagated to the front. Thus, the frontend was reporting that its cert had expired when in fact the backend's had. This turned out to be very confusing.

Note that I'm using the same certs for the front and back, and the front had expired and so I renewed it. I don't know if this had anything to do with it.

[wlallemand]

Some updates:

• show ssl crl-file on the CLI which help to have the status of a crl since 2.5.
• Two samples fetches were added ssl_bc_err and ssl_bc_err_str gives you more detail about SSL backend errors.