feat: apisix support QUIC+HTTP3

[lizj3624]

Issue description

cloudflare/quiche patch nginx-1.16 support quic+http3, I hope apisix support QUIC+HTTP3

[moonming]

great idea! I'm not familiar with quiche, how much would it cost to use it instead of openresty? In addition, nginx is officially merging the functions of QUIC+HTTP3? cc @membphis

[lizj3624]

I patch quiche to OpenResty-1.15.8，I found error (epoll_ctl(1, 53) failed (17: File exists)) when access_by_lua of ngx_lua module enabled，I gruess ngx_lua not support quic. ngx_lua issue

1. build env

nginx version: openresty/1.15.8.1 (quiche-09c0e37)
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC)
built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled
configure arguments: --prefix=/myngx/nginx --with-debug --with-cc-opt='-DNGX_LUA_USE_ASSERT -DNGX_LUA_ABORT_AT_PANIC -O2 -ggdb -O2 -D JD_NGX_SSL_HANDSHAKE_TIME -D JD_NGX_HTTP_UPSTREAM_RANDOM' --add-module=../ngx_devel_kit-0.3.1rc1 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.15 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.7 --with-ld-opt=-Wl,-rpath,/export/servers/myngx/luajit/lib --build=quiche-09c0e37 --with-pcre=/root/rpmbuild/BUILD/myngx-2.3-52.d375389/thirdparty/pcre-8.39 --with-pcre-jit --with-zlib=/root/rpmbuild/BUILD/myngx-2.3-52.d375389/thirdparty/zlib-1.2.8 --with-threads --with-http_auth_request_module --with-http_ssl_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_slice_module --add-module=/root/rpmbuild/BUILD/myngx-2.3-52.d375389/thirdparty/lua-ssl-nginx-module --add-module=/root/rpmbuild/BUILD/myngx-2.3-52.d375389/thirdparty/ngx_http_dyups_module --add-module=/root/rpmbuild/BUILD/myngx-2.3-52.d375389/thirdparty/ngx_http_sticky_module --with-stream --with-stream_ssl_module --with-http_v3_module --with-openssl=/root/myhome/quiche-cloudflare/deps/boringssl --with-quiche=/root/myhome/quiche-cloudflare --with-stream --with-stream_ssl_preread_module

2. some code ngx conf

http {
...
lua_max_running_timers 512;
lua_max_pending_timers 1024;

lua_package_path "$prefix/app/lib/?.lua;$prefix/app/etc/?.lua;$prefix/app/src/?.lua;$prefix/sbin/openresty/lualib/?.lua;;";
lua_package_cpath "$prefix/app/lib/?.so;$prefix/sbin/openresty/lualib/?.so;;";

lua_check_client_abort on;
lua_socket_log_errors off;
lua_http10_buffering off;
...

server {
    listen :443 quic reuseport;
    listen :443 ssl http2;
    ...
    access_by_lua_file my_waf.lua
    ...

    }
}

WAF module my_waf.lua

_M.waf_request = function ()
...ngx.req.get_body_file()..
local http_conn = http.new()
...http_conn.connect()...
...http_conn.request()...
...

end

3. error msg

2022/02/22 11:00:05 [alert] 27764#27764: *42474256 epoll_ctl(1, 36) failed (17: File exists), client: 1.114.102.56, server:quic.ngx.com, request: "POST /a?&t=1511418329196 HTTP/2.0", host: "quic.ngx.com"

[spacewander]

We have discussed it before. It seems that Nginx has developed its HTTP3 support instead of quiche. Since we need to catch up with the latest Nginx, it would be better to use Nginx's own HTTP3 implementation.

https://github.com/cloudflare/quiche/tree/master/nginx#building We can't go back to Nginx 1.16 to use quiche

[lizj3624]

@spacewander when apisix support QUIC+HTTP3？

[spacewander]

Once this feature is merged in Nginx and we catch up the latest version.

[membphis]

You can make a try: https://hg.nginx.org/nginx-quic/shortlog/quic + openresty @lizj3624

[lizj3624]

You can make a try: https://hg.nginx.org/nginx-quic/shortlog/quic + openresty @lizj3624

@membphis thx，from the test results，quiche's stability is better than nginx-quic, cloudflare‘s quiche has been running online for a long time，nginx-quic has not been merged into the master branch。

[spacewander]

Yes, but we are not Cloudflare. The Cloudflare-Nginx is a fork from an old Nginx version. We don't want to make APISIX hard to catch up with the latest Nginx.

[lizj3624]

1. OK，I understand apisix
2. official nginx plans merge quic to master before 2022，nginx h3 roadmap，but now is 2022.2，nginx new feat is too slow，alibaba，tencent，kuaishou LB supports quic，We are also testing cloudflare-quic nginx

[github-actions[bot]]

This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.

[Wlitsoft]

nginx-1.25.0 mainline version has been released, featuring experimental HTTP/3 support.

[gacktfly]

any plans for this？

[moonming]

any plans for this？

I think APISIX can be upgraded with the new version of NGINX

[gacktfly]

any plans for this？

I think APISIX can be upgraded with the new version of NGINX

great，cant wait to see it

[monkeyDluffy6017]

Please refer to https://github.com/apache/apisix/issues/9752

[NiuBlibing]

Hi, openresty 1.25.3.1 has been released, is it ready to upgrade it for apisix?

[monkeyDluffy6017]

Yeah, we will upgrade ASAP

[NiuBlibing]

Yeah, we will upgrade ASAP

@monkeyDluffy6017 Will it be released in 2024 Q1?

[monkeyDluffy6017]

@NiuBlibing I hope so

[zll600]

@monkeyDluffy6017 I want to do this, please assign to me

[SilentEntity]

@zll600, let's work together on this one. [co-work]

[zll600]

@SilentEntity Thank you. Here is my plan. I want to upgrade openresty-1.25.3.1 to make apisix support QUIC+HTTP3. I will be focusing on doing this for the next few days, if you have any ideas please let me know.

Step 1

• [ ] upgrade patch file in https://github.com/api7/ngx_multi_upstream_module:
  • [ ] https://github.com/api7/ngx_multi_upstream_module/pull/12
  • [ ] https://github.com/api7/ngx_multi_upstream_module/pull/13
• [ ] upgrade patch file in https://github.com/api7/apisix-nginx-module: https://github.com/api7/apisix-nginx-module/pull/83
• [ ] upgrade grpc-client-nginx-module: https://github.com/api7/grpc-client-nginx-module/pull/73
• [ ] upgrade wasm-nginx-module: https://github.com/api7/wasm-nginx-module/pull/162
• [ ] Upgrade apisix-runtime to use openresty-1.25.3.1: https://github.com/api7/apisix-build-tools/pull/367
• [ ] Upgrade apisix to use new version of apisix-runtime: https://github.com/apache/apisix/pull/10887

Step 2

• [ ] Provide quic configuration in apisix so that basic plug-ins can use quic: https://github.com/apache/apisix/pull/10970

Step 3

• [ ] Test quic’s compatibility under all plugins.

[NiuBlibing]

Congratulations! May I ask how soon the new version will be released?