Support for large CA lists via CLI `set ssl ca-file <cafile> <payload>`, limited by buffer size

[peanball]

Your Feature Request

Ideally, we would like to either:

• define a separate buffer size for the stats socket, which we could then tune to account for our largest expected (and limited) CA file size;
• or keep reading from the socket via buffer until the two newlines are received in payload mode (<< delimiter);
• or instruct HAProxy to read the data from a file on disk into memory;
• or append certificates to an existing CA file, e.g. via:

  add ssl ca-file <cafile> <payload>
  

  We could then loop through the individual PEM blocks and add those.

What are you trying to do?

We are setting a ca-file via the socket API / CLI. This requires the whole content of the CA file to be included in the request payload (delimited by <<) as part of the CLI request. In a multi-tenant system, our customers define the CAs they would like to add.

Adding some CA files via the socket failed, but worked fine via the file system. We narrowed down the issue to the pool buffer size (defined by tune.bufsize), which is also used for the stats socket connection. We are exploring to increase the buffer size but that has a global effect on all buffers, including HTTP request buffers, etc.

Increasing that size is not risk-free for us and ideally we would like to decouple the CA size from buffers used for other tasks in HAProxy.

Output of haproxy -vv

HAProxy version 2.5.4-e55ab42 2022/02/25 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2023.
Known bugs: http://www.haproxy.org/bugs/bugs-2.5.4.html
Running on: Linux 5.4.0-120-generic #136~18.04.1-Ubuntu SMP Fri Jun 10 18:00:44 UTC 2022 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=yes USE_STATIC_PCRE2=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1
  DEBUG   =

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL +THREAD +BACKTRACE -STATIC_PCRE +STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL +LUA +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC -PROMEX -MEMORY_PROFILING

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=4).
Built with OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.3
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Support for malloc_trim() is enabled.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.40 2022-04-14
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 7.5.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2       flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
            fcgi : mode=HTTP       side=BE        mux=FCGI     flags=HTX|HOL_RISK|NO_UPG
       <default> : mode=HTTP       side=FE|BE     mux=H1       flags=HTX
              h1 : mode=HTTP       side=FE|BE     mux=H1       flags=HTX|NO_UPG
       <default> : mode=TCP        side=FE|BE     mux=PASS     flags=
            none : mode=TCP        side=FE|BE     mux=PASS     flags=NO_UPG

Available services : none

Available filters :
        [SPOE] spoe
        [CACHE] cache
        [FCGI] fcgi-app
        [COMP] compression
        [TRACE] trace

[wlallemand]

You are indeed reaching a limitation of the current system, we have this problem with other commands but we split some of them into a transaction mechanism in order to bypass this.

Ideally we need a CLI which is capable of reading much bigger files but it will need some big changes, the easiest short-term solution would be to do a transaction system like we've done with the "crt-list". We already have everything we need with the "new ssl ca-file" and "commit ssl ca-file", so an "add ssl ca-file" would make sense.

[wlallemand]

I pushed a small PoC which basically does what you need, it's not finished but you can try this branch with these commands:

printf "set ssl ca-file ${testdir}/set_cafile_interCA1.crt <<\n$(cat ${testdir}/set_cafile_interCA1.crt)\n\n" | socat "${tmpdir}/h1/stats" -
printf "add ssl ca-file ${testdir}/set_cafile_interCA1.crt <<\n$(cat ${testdir}/set_cafile_interCA2.crt)\n\n" | socat "${tmpdir}/h1/stats" -

( It's from reg-tests/ssl/set_ssl_cafile.vtc )

[peanball]

@wlallemand thank you! I've tried it with adding some certificates and it works as intended, i.e. I can append certificates. I would need to make a few more tests, but conceptually this solves our problem.

Would you consider this a "shortcoming" of the ca-file editing feature introduced in HAProxy 2.5? I am trying to judge, if this would be something that would be back-ported to HAProxy 2.5 or would be a new feature for HAProxy 2.7+? This would inform our way forward in terms of waiting for a new patch release or planning mid-term workarounds.

One minor thing you may know: if you use add on an empty ca-file, i.e. you don't set any of the entries, HAProxy will segfault. I know this is a PoC, still wanted to provide what I found so it's not lost.

[wlallemand]

This is now available in master and will be in 2.7, I'm closing the ticket.