feature: new ngx_meta_lua_module (with multi-subsystems lua_shared_dict support)

[thibaultcha]

trafficstars

This PR introduces a new NGINX core module: ngx_meta_lua_module.

This new module aims to:

• Unify duplicated implementation between ngx_http_lua_module and ngx_stream_lua_module (I believe that many parts of both modules currently implemented via the tt2 templating system could eventually be rewritten more elegantly, simplifying lua-resty-core's codebase with regards to subsystems support, and also producing smaller binaries overall).
• Support the long-pending work item of sharing lua_shared_dict between the http {} and stream {} subsystems (with some code cleanup along the way).

ngx_meta_lua_module offers a new nginx.conf configuration block: lua {}.

New lua {} conf block

In the below example, we specify lua_shared_dict in the new lua {} block to share the allocated shm zone between all subsystems:

lua {
    lua_shared_dict dogs 1m;
}

http {
    lua_shared_dict cats 1m;

    server {
        location /t {
            content_by_lua_block {
                ngx.shared.dogs:get("foo") -- works
                ngx.shared.cats:get("foo") -- works
             }
        }
    }
}

stream {
    server {
        content_by_lua_block {
            ngx.shared.dogs:get("foo") -- works too!
            ngx.shared.cats:get("foo") -- does not work (http-only)
        }
    }
}

Currently, lua_shared_dict is the only directive supported inside of the lua {} configuration block. Many more could follow when settings could be applied globally to all subsystems' Lua VMs (e.g. lua_package_path, lua_max_pending_timers, etc...). In some cases, it would also make sense to only allow some Lua directives to be specified from within the lua {} block, e.g. lua_sa_restart.

Current state

For the time being, this PR is opened for feedback and reviews without an estimated timeline for merging.

That said, all tests are passing for:

• :heavy_check_mark: ngx_http_lua_module
• :heavy_check_mark: ngx_stream_lua_module
• :heavy_check_mark: lua-resty-core
• :heavy_check_mark: ngx_meta_lua_module (including HUP and Valgrind modes)

See also

For this module to work, it must be compiled with updated versions of ngx_http_lua_module, ngx_stream_lua_module, and lua-resty-core. See the following branches:

• https://github.com/thibaultcha/lua-nginx-module/tree/feat/ngx_meta_lua_module
• https://github.com/thibaultcha/stream-lua-nginx-module/tree/feat/ngx_meta_lua_module
• https://github.com/thibaultcha/lua-resty-core/tree/feat/ngx_meta_lua_module

The tt2 templates of this repositories have also been updated to support the ngx_meta_lua_module, and produce valid ngx_stream_lua_module/ngx_http_lua_module targets.

A list of topics requesting this feature:

• https://github.com/openresty/stream-lua-nginx-module/issues/25
• https://github.com/openresty/stream-lua-nginx-module/pull/36
• https://github.com/openresty/meta-lua-nginx-module/issues/24
• https://github.com/openresty/meta-lua-nginx-module/issues/41
• https://groups.google.com/forum/?hl=en#!searchin/openresty-en/lua_shared_dict|sort:date/openresty-en/883iv_w1v6U/bBVyNN0IBAAJ

TODOs

A non-exhaustive list:

• [ ] Add CI for the new ngx_meta_lua_module (test without http and without stream subsystems too)
• [ ] Debate the consequences of removing the public C API (which seems rather useless to me? unless it is used by someone out there, but if so what for?)
• [ ] Open sibling PRs for other modules (as noted in the above section)

[onlonely]

I need this function exactly,But not yet merged.

[stallion5632]

I need this function exactly,But not yet merged. @onlonely refer to https://github.com/stallion5632/apisix-nginx-module

this repository is a part of https://github.com/api7/apisix-nginx-module To realize the openresty/meta-lua-nginx-module#76

how to use base openresty official 1.19.9.1 version Follow the steps below:

cd apisix-nginx-module/patch ./patch.sh ThePathOfYourOpenRestySrcDirectory /configure --add-module=../apisix-nginx-module/src/meta make -j10

[geofflancaster]

Bump. Can this be merged?