trafficserver icon indicating copy to clipboard operation
trafficserver copied to clipboard
verified publisher icon apache

Use quiche as QUIC implementation

Open maskit opened this issue 3 years ago • 8 comments

This PR adds a build option (--with-quiche) and code to use quiche as QUIC implementation instead of the original our own QUIC implementation.

The code for the original implementation is kept for later use in case we want more flexibility. If you use BoringSSL or OpenSSL from quictls, the original one will be compiled but it won't be enabled until you explicitly configure ATS to use QUIC.

Also, this PR adds support for QUICv1 (0x00000001) and "h3", the official versions, and drops the support for draft-27. Support for draft-29 is kept for now like other applications do.

The code is compatible with quiche-0.14.0.

maskit avatar Jul 12 '22 07:07 maskit

@bneradt The instructions look almost correct. The build error should disappear if you checkout 0.14.0 (tag).

maskit avatar Jul 14 '22 23:07 maskit

@bneradt The instructions look almost correct. The build error should disappear if you checkout 0.14.0 (tag).

Ah, OK. Thanks. I was on master. I didn't realize that it had to be exactly 0.14.0, not no later than 0.14.0.

When I try with the 0.14.0 quiche tag, however, the configure doesn't succeed for all quiche configure checks, though:

checking checking for quiche includes in /home/bneradt/src/quiche/quiche... ok
configure: adding /home/bneradt/src/quiche/quiche/lib to RPATH
checking for quiche_connect in -lquiche... no

Is that expected? My build steps are the same as above, I just checkout tag 0.14.0 rather than master in quiche before doing the cargo build.

Thanks!

bneradt avatar Jul 15 '22 01:07 bneradt

Well, it doesn't have to be exactly 0.14.0 (0.14.1 would work), but they don't have 0.14.x branch and API compatibility is not guaranteed on master.

checking for quiche_connect in -lquiche... no

This is odd. I've been using 0.14.0 on a couple of environments and never seen it before. Can you share the config.log?

I've been able to build (and run) the branch successfully on macOS, Fedora 36, and CentOS 7. Any build error is unexpected, there might be compile warnings though.

maskit avatar Jul 15 '22 01:07 maskit

Can you share the config.log?

Sure thing. Here's the config.log. I reproduced this using the same docker steps I described above, with the addition of checking out 0.14.0 before running the cargo build command.

bneradt avatar Jul 15 '22 01:07 bneradt

configure:27926: checking checking for quiche includes in /root/src/quiche/quiche configure:27937: result: not found configure:27990: adding /root/src/quiche/quiche/lib to RPATH configure:28016: checking for quiche_connect in -lquiche configure:28041: cc -o conftest -D_GNU_SOURCE -DOPENSSL_NO_SSL_INTERN -I/root/src/quiche/quiche/include -L/root/src/quiche/quiche/lib conftest.c -lquiche -lpthread -ldl >&5 /usr/bin/ld: cannot find -lquiche

It seems like quiche.h and libquiche.so don't exist. Did cargo build succeed and generate the files? Just switching to 0.14.0 shouldn't cause the error.

maskit avatar Jul 15 '22 02:07 maskit

configure:27926: checking checking for quiche includes in /root/src/quiche/quiche configure:27937: result: not found configure:27990: adding /root/src/quiche/quiche/lib to RPATH configure:28016: checking for quiche_connect in -lquiche configure:28041: cc -o conftest -D_GNU_SOURCE -DOPENSSL_NO_SSL_INTERN -I/root/src/quiche/quiche/include -L/root/src/quiche/quiche/lib conftest.c -lquiche -lpthread -ldl >&5 /usr/bin/ld: cannot find -lquiche

It seems like quiche.h and libquiche.so don't exist. Did cargo build succeed and generate the files? Just switching to 0.14.0 shouldn't cause the error.

Yes, the cargo build succeeded and the files were generated:


# Reproduce the failure after the 0.14.0 build:

./configure --with-quiche=${HOME}/src/quiche/quiche --prefix=${HOME}/build/ts_quiche
...
checking checking for quiche includes in /root/src/quiche/quiche... ok
configure: adding /root/src/quiche/quiche/lib to RPATH
checking for quiche_connect in -lquiche... no
...

[root@e4ef3614f7c9 quiche]# ls ${HOME}/src/quiche/quiche/lib
build  deps  examples  incremental  libquiche.a  libquiche.d  libquiche.rlib  libquiche.so  quiche.pc
[root@e4ef3614f7c9 quiche]# ls ${HOME}/src/quiche/quiche/include
quiche.h

[root@e4ef3614f7c9 quiche]# cd ~/src/quiche/
[root@e4ef3614f7c9 quiche]# git show --quiet
commit 4d411c22413835f2d57f993d1c90c07813f803cd (HEAD, tag: 0.14.0)
Author: Alessandro Ghedini <[email protected]>
Date:   Mon May 23 11:57:20 2022 +0100

    0.14.0

I can then go back into the quiche tree, checkout master, re-run cargo, then rebuild trafficserver and the configure works:

./configure --with-quiche=${HOME}/src/quiche/quiche --prefix=${HOME}/build/ts_quiche
...
checking checking for quiche includes in /root/src/quiche/quiche... ok
configure: adding /root/src/quiche/quiche/lib to RPATH
checking for quiche_connect in -lquiche... yes
checking quiche.h usability... yes
checking quiche.h presence... yes
checking for quiche.h... yes

Then I can re-checkout quiche 0.14.0, rebuild via cargo, then reconfigure and it reproduces the configuration failure.

If you like, you can reproduce the issue easily via the docker commands I pasted above - just checkout the appropriate quiche version before the cargo build step.

Looking at the config.log, I notice this in the failed case:

configure:27926: checking checking for quiche includes in /root/src/quiche/quiche
configure:27934: result: ok
configure:27990: adding /root/src/quiche/quiche/lib to RPATH
configure:28016: checking for quiche_connect in -lquiche
configure:28041: cc -o conftest  -D_GNU_SOURCE -DOPENSSL_NO_SSL_INTERN -I/root/src/quiche/quiche/include -L/root/src/quiche/quiche/lib conftest.c -lquiche  -lpthread -ldl  >&5
/root/src/quiche/quiche/lib/libquiche.so: undefined reference to `X509_STORE_CTX_init'
/root/src/quiche/quiche/lib/libquiche.so: undefined reference to `EVP_hpke_aes_256_gcm'
/root/src/quiche/quiche/lib/libquiche.so: undefined reference to `EVP_HPKE_CTX_zero'
/root/src/quiche/quiche/lib/libquiche.so: undefined reference to `HRSS_encap'

It looks like the libquiche.so has symbol issues. On the other hand, when linking with quiche master:

configure:27926: checking checking for quiche includes in /root/src/quiche/quiche
configure:27934: result: ok
configure:27990: adding /root/src/quiche/quiche/lib to RPATH
configure:28016: checking for quiche_connect in -lquiche
configure:28041: cc -o conftest  -D_GNU_SOURCE -DOPENSSL_NO_SSL_INTERN -I/root/src/quiche/quiche/include -L/root/src/quiche/quiche/lib conftest.c -lquiche  -lpthread -ldl  >&5
configure:28041: $? = 0
configure:28050: result: yes
configure:28059: checking quiche.h usability
configure:28059: cc -c  -D_GNU_SOURCE -DOPENSSL_NO_SSL_INTERN -I/root/src/quiche/quiche/include conftest.c >&5
configure:28059: $? = 0
configure:28059: result: yes
configure:28059: checking quiche.h presence
configure:28059: cc -E -D_GNU_SOURCE -DOPENSSL_NO_SSL_INTERN -I/root/src/quiche/quiche/include conftest.c
configure:28059: $? = 0
configure:28059: result: yes
configure:28059: checking for quiche.h
configure:28059: result: yes

I did a git bisect on quiche and found that this is the commit on master after 0.14.0 that first builds a version of quiche that successfully passes ATS configuration checks:

1177c325f8288462299c26de5a94946288226b1b

commit 1177c325f8288462299c26de5a94946288226b1b (refs/bisect/bad)
Author: Junho Choi <[email protected]>
Date:   Tue Jun 21 03:14:44 2022 +0900

    ci: fixing MinGW cross build, adding to CI
    
    - Read target_env from CARGO_TARGET_* vars.
    - Override _WIN32_WINNT for building in MinGW.
    - Fix linking order (-lssl -lcrypto) for MinGW linker.
    - Add Windows + MinGW (x86_64-pc-windows-gnu)
      to github workflow (MinGW 32bit is still broken, not included).

Doing a quick experiment, it seems like in particular their fix to the order between ssl and crypto is what fixes things for us. That is, this patch in quiche results in an .so that successfully links for us:

diff --git a/quiche/src/build.rs b/quiche/src/build.rs
index 38f37e2..73f9f3f 100644
--- a/quiche/src/build.rs
+++ b/quiche/src/build.rs
@@ -235,13 +249,13 @@ fn main() {
         let build_dir = format!("{}/build/{}", bssl_dir, build_path);
         println!("cargo:rustc-link-search=native={}", build_dir);
 
-        println!("cargo:rustc-link-lib=static=crypto");
         println!("cargo:rustc-link-lib=static=ssl");
+        println!("cargo:rustc-link-lib=static=crypto");
     }
 
     if cfg!(feature = "boringssl-boring-crate") {
-        println!("cargo:rustc-link-lib=static=crypto");
         println!("cargo:rustc-link-lib=static=ssl");
+        println!("cargo:rustc-link-lib=static=crypto");
     }
 
     // MacOS: Allow cdylib to link with undefined symbols

It seems like we need to build against a version of quiche with that patch, if I'm not mistaken, and 0.14.0 will not pass our configuration check. At least not for Rocky (the new CentOS). Maybe other systems, like MacOS, is fine.

In any case, I'll continue with using the master version of quiche.

bneradt avatar Jul 15 '22 19:07 bneradt

Oh, I see. The failures I'm seeing using quiche master are because they changed the API of some of the functions we use in our ATS quiche branch since 0.14.0. Therefore the patch in this PR does not build against quiche master. That is annoying.

This will prevent us from testing this in CI since CI uses that ci.trafficserver.apache.org/ats/rockylinux:8 docker image with which I produced those build issues above. I'm not sure what our best path forward is.

@maskit : what do you think of rebasing this quiche branch patch to build against current quiche master instead of 0.14.0? Then we can build a version of quiche that works in CI with which we can build our ATS quiche branch.

Thank you for working on this.

bneradt avatar Jul 15 '22 19:07 bneradt

I added support for recent quiche. Now the code is compatible with the both 0.14.0 and current master (b841ca5) of quiche.

maskit avatar Jul 21 '22 05:07 maskit