lsquic lsquic connection not verifying server certificate

Hello

I've been experimenting with lsquic and trying to establish a connection using the following commands. I realized that the client is not verifying the server certificate during the connection.

build/bin/./http_server -L debug -c www.example.com,rsa/key_crt.pem,rsa/key_srv.pem -s 10.0.2.7:4433 -p /
build/bin/./http_client -L debug -C rsa/client -H www.example.com -s 10.0.2.7:4433 -p /

Likewise, the files: rsa/client/key_CA.pem, rsa/server/key_crt.pem, and rsa/server/key_srv.pem are in the proper directories to be referenced.

In http_client, I add this line after the verify_server_cert function definition: LSQ_DEBUG("!!! TESTING: verify_server_cert called");

I am wondering why the verify does not seem to be called and is there anything that I can modify to ensure that verification does happen? Attached are the two debug files.

Any help would be greatly appreciated.

Apr 26 '21 03:04 mraavi5

Only IETF QUIC (HTTP/3) supports that, so, please make sure you are testing HTTP/3, not Google QUIC.

Apr 26 '21 13:04 litespeedtech

Log shows h3-34 is used indeed. The certificate verification callback should be called by BoringSSL when handle the TLSv1.3 stream data, you may have to trace into that to find out why not called.

Apr 26 '21 13:04 litespeedtech

Hi,

To follow up on this issue, I was able to successfully trace the verification into BoringSSL and find the exact following line numbers that trigger an invalid verification. The tracing is done by adding a printf("!!!") to the BoringSSL code, which shows up on the terminal when I initiate a lsquic connection.

https://github.com/open-quantum-safe/boringssl/blob/master/crypto/x509/x509_vfy.c#L448 I found that line 448 of crypto/x509/x509_vfy.c is called, thus triggering the X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY error.
https://github.com/open-quantum-safe/boringssl/blob/master/crypto/x509/x509_vfy.c#L466 Following the previous error, this then triggers line 466 to be called, which calls goto end.
https://github.com/open-quantum-safe/boringssl/blob/master/crypto/x509/x509_vfy.c#L517 This sets the program counter to the end: label on line 517, which returns an error value of 0 with the variable bad_chain being toggled true. Meaning that the BoringSSL verification was unsuccessful.

What's concerning about this is that the same chain of events occurs when I use a tampered/bad certificate file; in every situation, the handshake is successful even though the x509_vfy.c file has an error. This leads me to doubt the error handling of lsquic, since the handshake is "successful" in all scenarios.

May 11 '21 20:05 mraavi5

Good finding. Can you show us the full call stack when X509_verify_cert() is called? There should be some error handling code at certain caller stack level to fail it. Since the handshake is handled by BoringSSL, we expect that the handshake to be failed in there and return the caller an error code. Maybe some misunderstanding how it works.

May 11 '21 20:05 litespeedtech

Thank you for the quick response. For the stack trace, I use the following code: https://www.gnu.org/software/libc/manual/html_node/Backtraces.html

I put the print_trace() function just after line 525 (https://github.com/open-quantum-safe/boringssl/blob/master/crypto/x509/x509_vfy.c#L525), it gives me the following output:

Obtained 10 stack frames.
/home/user/boringssl/build/crypto/libcrypto.so(+0x1639e1) [0x7f9cc08bd9e1]
/home/user/boringssl/build/crypto/libcrypto.so(X509_verify_cert+0xe3d) [0x7f9cc08be9d8]
/home/user/boringssl/build/ssl/libssl.so(+0x65ad2) [0x7f9cc0f05ad2]
/home/user/boringssl/build/ssl/libssl.so(+0x27c9c) [0x7f9cc0ec7c9c]
/home/user/boringssl/build/ssl/libssl.so(+0x7e91d) [0x7f9cc0f1e91d]
/home/user/boringssl/build/ssl/libssl.so(+0x7f2d1) [0x7f9cc0f1f2d1]
/home/user/boringssl/build/ssl/libssl.so(+0x2c3be) [0x7f9cc0ecc3be]
/home/user/boringssl/build/ssl/libssl.so(+0x2f7d2) [0x7f9cc0ecf7d2]
/home/user/boringssl/build/ssl/libssl.so(+0x28b53) [0x7f9cc0ec8b53]
/home/user/boringssl/build/ssl/libssl.so(SSL_do_handshake+0xaf) [0x7f9cc0ef4546]

I modified the print_trace() function to show the last 100 call stack items. It only gave 31. Here are the results:

Obtained 31 stack frames.
/home/user/boringssl/build/crypto/libcrypto.so(+0x1639e7) [0x7f05b355a9e7]
/home/user/boringssl/build/crypto/libcrypto.so(X509_verify_cert+0xe3d) [0x7f05b355ba05]
/home/user/boringssl/build/ssl/libssl.so(+0x65ad2) [0x7f05b3ba2ad2]
/home/user/boringssl/build/ssl/libssl.so(+0x27c9c) [0x7f05b3b64c9c]
/home/user/boringssl/build/ssl/libssl.so(+0x7e91d) [0x7f05b3bbb91d]
/home/user/boringssl/build/ssl/libssl.so(+0x7f2d1) [0x7f05b3bbc2d1]
/home/user/boringssl/build/ssl/libssl.so(+0x2c3be) [0x7f05b3b693be]
/home/user/boringssl/build/ssl/libssl.so(+0x2f7d2) [0x7f05b3b6c7d2]
/home/user/boringssl/build/ssl/libssl.so(+0x28b53) [0x7f05b3b65b53]
/home/user/boringssl/build/ssl/libssl.so(SSL_do_handshake+0xaf) [0x7f05b3b91546]
/home/user/lsquic/build/bin/./http_client(+0x38e2e) [0x55e817f7ee2e]
/home/user/lsquic/build/bin/./http_client(+0x3ce8d) [0x55e817f82e8d]
/home/user/lsquic/build/bin/./http_client(+0x3d3c3) [0x55e817f833c3]
/home/user/lsquic/build/bin/./http_client(+0xc96fd) [0x55e81800f6fd]
/home/user/lsquic/build/bin/./http_client(lsquic_stream_dispatch_read_events+0x42) [0x55e818010125]
/home/user/lsquic/build/bin/./http_client(+0x6cbaa) [0x55e817fb2baa]
/home/user/lsquic/build/bin/./http_client(+0x6cc0f) [0x55e817fb2c0f]
/home/user/lsquic/build/bin/./http_client(+0x7194d) [0x55e817fb794d]
/home/user/lsquic/build/bin/./http_client(+0x72499) [0x55e817fb8499]
/home/user/lsquic/build/bin/./http_client(+0x7451a) [0x55e817fba51a]
/home/user/lsquic/build/bin/./http_client(+0x754f7) [0x55e817fbb4f7]
/home/user/lsquic/build/bin/./http_client(+0x75657) [0x55e817fbb657]
/home/user/lsquic/build/bin/./http_client(+0x4176e) [0x55e817f8776e]
/home/user/lsquic/build/bin/./http_client(lsquic_engine_packet_in+0x644) [0x55e817f8d05c]
/home/user/lsquic/build/bin/./http_client(+0x2f759) [0x55e817f75759]
/home/user/lsquic/build/bin/./http_client(+0x1197c4) [0x55e81805f7c4]
/home/user/lsquic/build/bin/./http_client(event_base_loop+0x52f) [0x55e8180601ff]
/home/user/lsquic/build/bin/./http_client(prog_run+0xc6) [0x55e817f7382d]
/home/user/lsquic/build/bin/./http_client(main+0xa54) [0x55e817f723ee]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f05b2a6abf7]
/home/user/lsquic/build/bin/./http_client(_start+0x2a) [0x55e817f6db3a]

Since I'm using the compiled http_client and http_server, the stack trace is in reference to the actual assembly code. I'll look for a better way to get the call stack, maybe wrapping the functions with print statements... if you have any ideas I'd love to try them.

May 11 '21 21:05 mraavi5

That's hard to decrypt. :-)
An easier way is to compile BoringSSL as well as lsquic with debug symbol. then load http_client binary under gdb, set breakpoint to x509_vfy.c:466, run, print backtrace with bt after it hits the breakpoint.

May 11 '21 22:05 litespeedtech

Thank you for the directions. The following contains the requested information:

user@user:~$ gdb --args ~/lsquic/build/bin/./http_client -C ~/lsquic/my_rsa_cert_directory -H www.example.com -s 10.0.2.9:4433 -p /

 

GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/user/lsquic/build/bin/./http_client...done.

 


(gdb) # COMMENT: The last return of the X509_verify_cert function: (https://github.com/open-quantum-safe/boringssl/blob/master/crypto/x509/x509_vfy.c#L526).

 

(gdb) break x509_vfy.c:526

 

No source file named x509_vfy.c.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (x509_vfy.c:526) pending.

 


(gdb) run

 

Starting program: /home/user/lsquic/build/bin/http_client -C /home/user/lsquic/my_rsa_cert_directory -H www.example.com -s 10.0.2.9:4433 -p /
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

 

Breakpoint 1, X509_verify_cert (ctx=0x7fffffffcdc0)
    at ../crypto/x509/x509_vfy.c:526
526        return ok;

 


(gdb) bt

 

#0  X509_verify_cert (ctx=0x7fffffffcdc0) at ../crypto/x509/x509_vfy.c:526
#1  0x00007ffff74a1ab2 in bssl::ssl_crypto_x509_session_verify_cert_chain (
    session=0x555555a86958, hs=0x555555a82738, out_alert=0x7fffffffcee8 "P")
    at ../ssl/ssl_x509.cc:403
#2  0x00007ffff7463c7c in bssl::ssl_verify_peer_cert (hs=0x555555a82738)
    at ../ssl/handshake.cc:354
#3  0x00007ffff74ba8fd in bssl::do_read_server_certificate_verify (
    hs=0x555555a82738) at ../ssl/tls13_client.cc:639
#4  0x00007ffff74bb2b1 in bssl::tls13_client_handshake (hs=0x555555a82738)
    at ../ssl/tls13_client.cc:852
#5  0x00007ffff746839e in bssl::do_tls13 (hs=0x555555a82738)
    at ../ssl/handshake_client.cc:795
#6  0x00007ffff746b7b2 in bssl::ssl_client_handshake (hs=0x555555a82738)
    at ../ssl/handshake_client.cc:1767
#7  0x00007ffff7464b33 in bssl::ssl_run_handshake (hs=0x555555a82738, 
    out_early_return=0x7fffffffd0ab) at ../ssl/handshake.cc:689
#8  0x00007ffff7490526 in SSL_do_handshake (ssl=0x555555a83c58)
    at ../ssl/ssl_lib.cc:889
#9  0x000055555558cdba in iquic_esfi_handshake (enc_sess=0x555555ad0010)
    at ../src/liblsquic/lsquic_enc_sess_ietf.c:1905
#10 0x0000555555590e19 in iquic_esfi_shake_stream (sess=0x555555ad0010, 
    stream=0x555555a87520, what=0x55555567d24e "on_read")
    at ../src/liblsquic/lsquic_enc_sess_ietf.c:3160
#11 0x000055555559134f in chsk_ietf_on_read (stream=0x555555a87520, 
    ctx=0x555555ad0010) at ../src/liblsquic/lsquic_enc_sess_ietf.c:3250
#12 0x000055555561d689 in stream_dispatch_read_events_loop (
    stream=0x555555a87520) at ../src/liblsquic/lsquic_stream.c:2170
#13 0x000055555561e0b1 in lsquic_stream_dispatch_read_events (
    stream=0x555555a87520) at ../src/liblsquic/lsquic_stream.c:2381
#14 0x00005555555c0b36 in process_crypto_frame_client (conn=0x555555acc0c0, 
    packet_in=0x555555ab2080, 
    p=0x555555a8924a "\006D\266A7c\001\252\271\r\327:", len=316)
    at ../src/liblsquic/lsquic_full_conn_ietf.c:5670
---Type <return> to continue, or q <return> to quit---
#15 0x00005555555c0b9b in process_crypto_frame (conn=0x555555acc0c0, 
    packet_in=0x555555ab2080, 
    p=0x555555a8924a "\006D\266A7c\001\252\271\r\327:", len=316)
    at ../src/liblsquic/lsquic_full_conn_ietf.c:5684
#16 0x00005555555c58d9 in process_packet_frame (conn=0x555555acc0c0, 
    packet_in=0x555555ab2080, 
    p=0x555555a8924a "\006D\266A7c\001\252\271\r\327:", len=316)
    at ../src/liblsquic/lsquic_full_conn_ietf.c:6721
#17 0x00005555555c6425 in parse_regular_packet (conn=0x555555acc0c0, 
    packet_in=0x555555ab2080) at ../src/liblsquic/lsquic_full_conn_ietf.c:6878
#18 0x00005555555c84a6 in process_regular_packet (conn=0x555555acc0c0, 
    packet_in=0x555555ab2080) at ../src/liblsquic/lsquic_full_conn_ietf.c:7392
#19 0x00005555555c9483 in process_incoming_packet_fast (conn=0x555555acc0c0, 
    packet_in=0x555555ab2080) at ../src/liblsquic/lsquic_full_conn_ietf.c:7630
#20 0x00005555555c95e3 in ietf_full_conn_ci_packet_in (lconn=0x555555acc0c0, 
    packet_in=0x555555ab2080) at ../src/liblsquic/lsquic_full_conn_ietf.c:7659
#21 0x00005555555956fa in process_packet_in (engine=0x555555a8a550, 
    packet_in=0x555555ab2080, ppstate=0x7fffffffd5d0, sa_local=0x555555ac26a0, 
    sa_peer=0x555555ac7430, peer_ctx=0x555555a82360, packet_in_size=358)
    at ../src/liblsquic/lsquic_engine.c:1572
#22 0x000055555559afe8 in lsquic_engine_packet_in (engine=0x555555a8a550, 
    packet_in_data=0x7ffff7f92b3e "\301\377", packet_in_size=407, 
    sa_local=0x555555ac26a0, sa_peer=0x555555ac7430, peer_ctx=0x555555a82360, 
    ecn=0) at ../src/liblsquic/lsquic_engine.c:3077
#23 0x00005555555836e5 in read_handler (fd=6, flags=2, ctx=0x555555a82360)
    at ../bin/test_common.c:752
#24 0x000055555566d744 in event_process_active_single_queue ()
#25 0x000055555566e17f in event_base_loop ()
#26 0x00005555555817b9 in prog_run (prog=0x7fffffffd9d0) at ../bin/prog.c:603
#27 0x000055555558037a in main (argc=9, argv=0x7fffffffdf18)
    at ../bin/http_client.c:1856
(gdb) 

 


(gdb) info locals

 

x = 0x555555a879c8
xtmp = 0x0
xtmp2 = 0x555555aceb58
chain_ss = 0x0
bad_chain = 1
param = 0x555555a88478
depth = 100
i = 1
ok = 0
num = 1
j = 0
retry = 0
trust = 3
cb = 0x7ffff6abf829 <null_callback>
sktmp = 0x555555ad2e48
err = 0

 


(gdb) quit

May 14 '21 21:05 mraavi5

Good. that is helpful. As you can see, lsquic call SSL_do_handshake() at

#9  0x000055555558cdba in iquic_esfi_handshake (enc_sess=0x555555ad0010)
    at ../src/liblsquic/lsquic_enc_sess_ietf.c:1905

and check the return and call SSL_get_error(), handle the handshake error. Looks SSL_do_handshake() may not return an error, you can set break point there, check the return value.

May 14 '21 21:05 litespeedtech

Interesting, thank you.

Here is SSL_do_handshake():

(gdb) break ssl_lib.cc:SSL_do_handshake

 

No source file named ssl_lib.cc.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (ssl_lib.cc:SSL_do_handshake) pending.

 

(gdb) run

 

Starting program: /home/user/lsquic/build/bin/http_client -C /home/user/lsquic/my_rsa_cert_directory -H www.example.com -s 10.0.2.9:4433 -p /
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

 

Breakpoint 1, SSL_do_handshake (ssl=0x555555a83c58) at ../ssl/ssl_lib.cc:873
873 int SSL_do_handshake(SSL *ssl) {

 

(gdb) bt

 

#0  SSL_do_handshake (ssl=0x555555a83c58) at ../ssl/ssl_lib.cc:873
#1  0x000055555558cdba in iquic_esfi_handshake (enc_sess=0x555555ad0010)
    at ../src/liblsquic/lsquic_enc_sess_ietf.c:1905
#2  0x0000555555590e19 in iquic_esfi_shake_stream (sess=0x555555ad0010, 
    stream=0x555555acec20, what=0x55555567d304 "on_write")
    at ../src/liblsquic/lsquic_enc_sess_ietf.c:3160
#3  0x0000555555591605 in chsk_ietf_on_write (stream=0x555555acec20, 
    ctx=0x555555ad0010) at ../src/liblsquic/lsquic_enc_sess_ietf.c:3311
#4  0x000055555561dbd2 in stream_dispatch_write_events_loop (
    stream=0x555555acec20) at ../src/liblsquic/lsquic_stream.c:2291
#5  0x000055555561e179 in lsquic_stream_dispatch_write_events (
    stream=0x555555acec20) at ../src/liblsquic/lsquic_stream.c:2415
#6  0x00005555555bb31e in process_crypto_stream_write_events (
    conn=0x555555acc0c0) at ../src/liblsquic/lsquic_full_conn_ietf.c:4305
#7  0x00005555555cb78b in ietf_full_conn_ci_tick (lconn=0x555555acc0c0, 
    now=19670126159) at ../src/liblsquic/lsquic_full_conn_ietf.c:8302
#8  0x000055555559a162 in process_connections (engine=0x555555a8a550, 
    next_conn=0x555555596fcd <conn_iter_next_tickable>, now=19670126159)
    at ../src/liblsquic/lsquic_engine.c:2863
#9  0x000055555559753e in lsquic_engine_process_conns (engine=0x555555a8a550)
    at ../src/liblsquic/lsquic_engine.c:2049
#10 0x000055555558153a in prog_process_conns (prog=0x7fffffffd9d0)
    at ../bin/prog.c:540
---Type <return> to continue, or q <return> to quit---
#11 0x0000555555581097 in prog_connect (prog=0x7fffffffd9d0, sess_resume=0x0, 
    sess_resume_len=0) at ../bin/prog.c:408
#12 0x000055555557c219 in create_connections (client_ctx=0x7fffffffd930)
    at ../bin/http_client.c:272
#13 0x000055555558034a in main (argc=9, argv=0x7fffffffdf18)
    at ../bin/http_client.c:1852
(gdb) 

 

(gdb) info locals

 

hs = 0x7fffffffae90
early_return = false
ret = 0

 

(gdb) quit

Then for SSL_get_error(), it returns on line 1349 (https://github.com/open-quantum-safe/boringssl/blob/master/ssl/ssl_lib.cc#L1349) with the error code SSL_ERROR_WANT_READ.

The backtrace for that one is:

(gdb) bt
#0  SSL_get_error (ssl=0x555555a83c58, ret_code=-1) at ../ssl/ssl_lib.cc:1349
#1  0x000055555558cde8 in iquic_esfi_handshake (enc_sess=0x555555ad0010)
    at ../src/liblsquic/lsquic_enc_sess_ietf.c:1908
#2  0x0000555555590e19 in iquic_esfi_shake_stream (sess=0x555555ad0010, 
    stream=0x555555acec20, what=0x55555567d304 "on_write")
    at ../src/liblsquic/lsquic_enc_sess_ietf.c:3160
#3  0x0000555555591605 in chsk_ietf_on_write (stream=0x555555acec20, 
    ctx=0x555555ad0010) at ../src/liblsquic/lsquic_enc_sess_ietf.c:3311
#4  0x000055555561dbd2 in stream_dispatch_write_events_loop (
    stream=0x555555acec20) at ../src/liblsquic/lsquic_stream.c:2291
#5  0x000055555561e179 in lsquic_stream_dispatch_write_events (
    stream=0x555555acec20) at ../src/liblsquic/lsquic_stream.c:2415
#6  0x00005555555bb31e in process_crypto_stream_write_events (
    conn=0x555555acc0c0) at ../src/liblsquic/lsquic_full_conn_ietf.c:4305
#7  0x00005555555cb78b in ietf_full_conn_ci_tick (lconn=0x555555acc0c0, 
    now=23455753968) at ../src/liblsquic/lsquic_full_conn_ietf.c:8302
#8  0x000055555559a162 in process_connections (engine=0x555555a8a550, 
    next_conn=0x555555596fcd <conn_iter_next_tickable>, now=23455753968)
    at ../src/liblsquic/lsquic_engine.c:2863
#9  0x000055555559753e in lsquic_engine_process_conns (engine=0x555555a8a550)
    at ../src/liblsquic/lsquic_engine.c:2049
#10 0x000055555558153a in prog_process_conns (prog=0x7fffffffd9d0)
    at ../bin/prog.c:540
---Type <return> to continue, or q <return> to quit---
#11 0x0000555555581097 in prog_connect (prog=0x7fffffffd9d0, sess_resume=0x0, 
    sess_resume_len=0) at ../bin/prog.c:408
#12 0x000055555557c219 in create_connections (client_ctx=0x7fffffffd930)
    at ../bin/http_client.c:272
#13 0x000055555558034a in main (argc=9, argv=0x7fffffffdf18)
    at ../bin/http_client.c:1852
(gdb)

Which means:

// SSL_ERROR_WANT_READ indicates the operation failed attempting to read from // the transport. The caller may retry the operation when the transport is ready // for reading.

There must be some kind of difficulty in identifying where to read the certificate file?

Why does the -C parameter in http_client want a directory rather than the more precise .pem file path?

May 16 '21 18:05 mraavi5

SSL_ERROR_WANT_READ is not a real/fatal error, just the SSL need to read more data to complete the handshake. All SSL handshake will go through this to complete the handshake. Looks it mask the the error from certificate verification failure.

May 19 '21 14:05 litespeedtech

Hi, thanks for looking at it. Since the SSL_ERROR_WANT_READ is not the direct cause, I'm thinking that the transport may never be ready to read the certificate file because the file path may have some issues.

Is the file path specified by -C a relative path? Or is there a recommended storage location for the certificate file?

Jun 01 '21 20:06 mraavi5

Sorry for resurrecting this, but what I've found out may benefit someone who stumbles upon this thread via a search engine. I know the above discussion helped me, so thank you. Oh and to answer the unanswered question, the path specified by -C is passed unaltered to opendir(). So yes, it is relative to your current working directory, not relative to the program being ran. To be safe though, you could always provide an absolute path. Any *.pem file in that directory will be added to the certificate store for the SSL context being configured.

I'm not running the example http_client.c, just using it as a reference for an implementation of my own client. So whether this is a "bug" or "working as intended" I haven't bothered to find out...

When creating a client (not sure about server sorry), if one provides a "get SSL context" callback when creating an engine, then the "verify certificate" callback is ignored.

For example:

struct lsquic_engine_api eapi = {
  // init other stuff...

  .ea_get_ssl_ctx = get_ssl_ctx,  // user defined callback

  // this is ignored if
  // - ea_get_ssl_ctx is not NULL
  // - or if the ea_get_ssl_ctx callback does not return NULL
  .ea_verify_cert = verify_server_cert,  // user defined callback
};

This makes sense because if one is providing their own configured SSL context, then it is logical to presume the manually configured SSL context is set up to do the necessary things, such as verifying the server certificate is trusted by the client.

In the code, this can be seen here:

v3.1.3 lsquic_enc_sess_ietf:886
master (might be a moving target) lsquic_enc_sess_ietf:886

Sep 29 '22 00:09 nknowles88