xquic
xquic copied to clipboard
[Bug]: xquic无法支持babaSSL的async_job(协程)导致QAT加速失效
What happened?
xquic调用babaSSL实现的tls1.3,babaSSL支持async_job,也就是异步SSL,这样的话把加解密等耗时操作卸载到QAT后,job能立刻返回,cpu还能处理其他请求,可以大幅提升性能。但是实际测试下来xquic不支持babaSSL的异步job。比如xquic会调用SSL_do_handshake发起握手,在发起握手前可以设置ssl的mode是异步的,这样SSL_do_handshake的返回结果是SSL_ERROR_WANT_ASYNC,但是xquic直接当错误处理了,从代码角度未看到有对async_job的支持。
Steps To Reproduce
使用tengine+xquic+tongsuo搭建HTTP3的server环境,同时使用QAT硬件对加解密算法进行加速,但是测试效果发现使用QAT硬件无法提升HTTPS的性能。 大概分析了握手过程中的调用栈,原因应该是xquic不支持async_mode导致的。 Thread 1 "tengine" hit Breakpoint 1, qat_engine_ecdh_compute_key (out=out@entry=0x7fff4fcf0b18, outlen=outlen@entry=0x7fff4fcf0b20, pub_key=0x55a8bd93d870, ecdh=0x55a8bd92ff30) at qat_hw_ec.c:795 795 { (gdb) bt #0 qat_engine_ecdh_compute_key (out=out@entry=0x7fff4fcf0b18, outlen=outlen@entry=0x7fff4fcf0b20, pub_key=0x55a8bd93d870, ecdh=0x55a8bd92ff30) at qat_hw_ec.c:795 https://github.com/alibaba/tengine/pull/1 0x00007fcdb725a43f in QAT_ECDH_compute_key (out=out@entry=0x55a8bd92ba10, outlen=, outlen@entry=32, pub_key=, eckey=eckey@entry=0x55a8bd92ff30, KDF=KDF@entry=0x0) at qat_prov_ecdh.c:582 https://github.com/alibaba/tengine/pull/2 0x00007fcdb725a964 in qat_keyexch_ecdh_plain_derive (outlen=, psecretlen=0x7fff4fcf0cb0, secret=0x55a8bd92ba10 "\233\020\031\347\255U", vpecdhctx=0x55a8bd95af20) at qat_prov_ecdh.c:666 https://github.com/alibaba/tengine/pull/3 qat_keyexch_ecdh_derive (vpecdhctx=0x55a8bd95af20, secret=0x55a8bd92ba10 "\233\020\031\347\255U", psecretlen=0x7fff4fcf0cb0, outlen=) at qat_prov_ecdh.c:763 https://github.com/alibaba/tengine/pull/4 0x00007fcdb79aa2e2 in ssl_derive (s=s@entry=0x55a8bd964d60, privkey=, pubkey=pubkey@entry=0x55a8bd8f6d60, gensecret=gensecret@entry=1) at ssl/s3_lib.c:4125 https://github.com/alibaba/tengine/pull/5 0x00007fcdb79eae7d in tls_construct_stoc_key_share (s=0x55a8bd964d60, pkt=0x7fff4fcf0e30, context=, x=, chainidx=) at ssl/statem/extensions_srvr.c:1779 https://github.com/alibaba/tengine/pull/6 0x00007fcdb79e0eac in tls_construct_extensions (s=s@entry=0x55a8bd964d60, pkt=pkt@entry=0x7fff4fcf0e30, context=512, x=x@entry=0x0, chainidx=chainidx@entry=0) at ssl/statem/extensions.c:881 https://github.com/alibaba/tengine/pull/7 0x00007fcdb79facb2 in tls_construct_server_hello (s=0x55a8bd964d60, pkt=0x7fff4fcf0e30) at ssl/statem/statem_srvr.c:2560 https://github.com/alibaba/tengine/pull/8 0x00007fcdb79ec49d in write_state_machine (s=0x55a8bd964d60) at ssl/statem/statem.c:890 https://github.com/alibaba/tengine/pull/9 state_machine (s=0x55a8bd964d60, server=1) at ssl/statem/statem.c:482 https://github.com/alibaba/tengine/pull/10 0x00007fcdb79b8a08 in SSL_do_handshake (s=0x55a8bd964d60) at ssl/ssl_lib.c:4075 https://github.com/alibaba/tengine/issues/11 0x00007fcdb7b955e1 in xqc_ssl_do_handshake (ssl=0x55a8bd964d60) at /home/zhoubin/code/quic/xquic/src/tls/babassl/xqc_ssl_if.c:119 https://github.com/alibaba/tengine/pull/12 0x00007fcdb7b921af in xqc_tls_do_handshake (tls=0x55a8bd913480) at /home/zhoubin/code/quic/xquic/src/tls/xqc_tls.c:394 https://github.com/alibaba/tengine/pull/13 0x00007fcdb7b928da in xqc_tls_process_crypto_data (tls=0x55a8bd913480, level=XQC_ENC_LEV_INIT, crypto_data=0x55a8bd8e6f60 "\001", data_len=559) at /home/zhoubin/code/quic/xquic/src/tls/xqc_tls.c:568 https://github.com/alibaba/tengine/pull/14 0x00007fcdb7b630d0 in xqc_read_crypto_stream (stream=0x55a8bd95c2c0) at /home/zhoubin/code/quic/xquic/src/transport/xqc_stream.c:729 https://github.com/alibaba/tengine/pull/15 0x00007fcdb7b76cd2 in xqc_process_crypto_frame (conn=0x55a8bd913680, packet_in=0x7fff4fcf1190) at /home/zhoubin/code/quic/xquic/src/transport/xqc_frame.c:559 https://github.com/alibaba/tengine/pull/16 0x00007fcdb7b756b2 in xqc_process_frames (conn=0x55a8bd913680, packet_in=0x7fff4fcf1190) at /home/zhoubin/code/quic/xquic/src/transport/xqc_frame.c:224 https://github.com/alibaba/tengine/pull/17 0x00007fcdb7b7497c in xqc_packet_decrypt_single (c=0x55a8bd913680, packet_in=0x7fff4fcf1190) at /home/zhoubin/code/quic/xquic/src/transport/xqc_packet.c:183 https://github.com/alibaba/tengine/pull/18 0x00007fcdb7b74b35 in xqc_packet_process_single (c=0x55a8bd913680, packet_in=0x7fff4fcf1190) at /home/zhoubin/code/quic/xquic/src/transport/xqc_packet.c:226 https://github.com/alibaba/tengine/pull/19 0x00007fcdb7b4efbf in xqc_conn_process_packet (c=0x55a8bd913680, packet_in_buf=0x55a8bcecdbc8 <packet+232> <incomplete sequence \303>, packet_in_size=1216, recv_time=1706106787763823) at /home/zhoubin/code/quic/xquic/src/transport/xqc_conn.c:3528 https://github.com/alibaba/tengine/pull/20 0x00007fcdb7b4112f in xqc_engine_packet_process (engine=0x55a8bd8b5590, packet_in_buf=0x55a8bcecdbc8 <packet+232> <incomplete sequence \303>, packet_in_size=1216, local_addr=0x55a8bcecdb54 <packet+116>, local_addrlen=16, peer_addr=0x55a8bcecdae0 , peer_addrlen=16, recv_time=1706106787763823, user_data=0x7fcdb48aa010) at /home/zhoubin/code/quic/xquic/src/transport/xqc_engine.c:1227 https://github.com/alibaba/tengine/pull/21 0x000055a8bce4995e in ngx_xquic_dispatcher_process_packet (c=c@entry=0x7fcdb48aa010, packet=packet@entry=0x55a8bcecdae0 ) at modules/ngx_http_xquic_module/ngx_xquic_recv.c:403 https://github.com/alibaba/tengine/pull/22 0x000055a8bce49ad5 in ngx_xquic_event_recv (ev=0x55a8bd885570) at modules/ngx_http_xquic_module/ngx_xquic_recv.c:312 https://github.com/alibaba/tengine/pull/23 0x000055a8bcdceca3 in ngx_epoll_process_events (cycle=, timer=, flags=) at src/event/modules/ngx_epoll_module.c:972 https://github.com/alibaba/tengine/pull/24 0x000055a8bcdc2c1e in ngx_process_events_and_timers (cycle=cycle@entry=0x55a8bd85eb90) at src/event/ngx_event.c:284 https://github.com/alibaba/tengine/pull/25 0x000055a8bcdcdf44 in ngx_single_process_cycle (cycle=cycle@entry=0x55a8bd85eb90) at src/os/unix/ngx_process_cycle.c:336 https://github.com/alibaba/tengine/pull/26 0x000055a8bcda0dd1 in main (argc=, argv=) at src/core/nginx.c:416 以上是抓取的调用栈,结合代码分析,所有函数的调用过程都是同步的,也就是说需要等到QAT完成加解密之后SSL_do_handshake才会返回,由于tengine只有一个线程,在硬件加解密期间无法处理其他的连接请求,从而导致性能无法提升。
非常非常希望tengine+xquic可以支持tongsuo的异步ssl,这样使用如QAT加速卡就能大幅提升性能了
Relevant log output
No response