[feature or help] `run_worker_thread` has too strong limitation

[suikabreaker]

trafficstars

As document says, we are not allowed to use any ngx_lua feature in run_worker_thread calls. I guess it's because thread has no request context to do those calls, but is it really necessary to forbid all ngx.* APIs?

For example:

1. string manipulate functions, including regex/encode/decode;
2. nginx enviroment, like ngx.get_phase() config.*
3. ngx.log()
4. shared memory ngx.shared

Are those calls allowed but not documented, or for some reason they're not supported, or they can be and will be supported?

[suikabreaker]

Take ngx.timer.at as reference:

A lot of the Lua APIs for Nginx are enabled in the context of the timer callbacks, like stream/datagram cosockets (ngx.socket.tcp and ngx.socket.udp), shared memory dictionaries (ngx.shared.DICT), user coroutines (coroutine.*), user "light threads" (ngx.thread.*), ngx.exit, ngx.now/ngx.time, ngx.md5/ngx.sha1_bin, are all allowed. But the subrequest API (like ngx.location.capture), the ngx.req.* API, the downstream output API (like ngx.say, ngx.print, and ngx.flush) are explicitly disabled in this context.

[suikabreaker]

And what makes it disabled in header_filter_by_lua body_filter_by_lua?

[suikabreaker]

And about the API call arguments, how should I call function defined in the same module which calls the run_worker_thread

[suikabreaker]

@zhuizhuhaomeng @kingluo @doujiang24

[doujiang24]

Are those calls allowed but not documented, or for some reason they're not supported, or they can be and will be supported?

1. They are not allowed in the current implementation.
2. But many ngx.* APIs can be supported, just not implemented yet. patches welcome!

[doujiang24]

And about the API call arguments, how should I call function defined in the same module which calls the run_worker_thread

I can not understand you, can you give a simple sample?

[suikabreaker]

And about the API call arguments, how should I call function defined in the same module which calls the run_worker_thread

I can not understand you, can you give a simple sample?

-- a.lua
function _M.func()
-- ...
end
function _M.on_access()
    ngx.run_worker_thread("my_thread_pool", "a", "func")
end

[doujiang24]

And about the API call arguments, how should I call function defined in the same module which calls the run_worker_thread

I can not understand you, can you give a simple sample?

-- a.lua
function _M.func()
-- ...
end
function _M.on_access()
    ngx.run_worker_thread("my_thread_pool", "a", "func")
end

@suikabreaker Have you tried the sample? Do you got an error or not?

[kingluo]

1. string manipulate functions, including regex/encode/decode;
2. nginx enviroment, like ngx.get_phase() config.*
3. ngx.log()
4. shared memory ngx.shared

The ngx.get_phase(), ngx.log(), ngx.shared are bound to the request, so it's impossible to use in a separate worker thread.

Now I tried to add the below APIs to run_worker_thread():

        /* inject API from C */
        lua_createtable(L, 0 /* narr */, 113 /* nrec */);    /* ngx.* */

        ngx_http_lua_inject_string_api(vm);
        ngx_http_lua_inject_config_api(vm);
        lua_setglobal(vm, "ngx");

        /* inject API via ffi */
        lua_getglobal(vm, "require");
        lua_pushstring(vm, "resty.core.regex");
        lua_pcall(vm, 1, 0, 0);

        lua_getglobal(vm, "require");
        lua_pushstring(vm, "resty.core.time");
        lua_pcall(vm, 1, 0, 0);

        lua_getglobal(vm, "require");
        lua_pushstring(vm, "resty.core.hash");
        lua_pcall(vm, 1, 0, 0);

        lua_getglobal(vm, "require");
        lua_pushstring(vm, "resty.core.base64");
        lua_pcall(vm, 1, 0, 0);

I am testing these APIs, and I would create a PR later.

[doujiang24]

The ngx.get_phase(), ngx.log(), ngx.shared are bound to the request, so it's impossible to use in a separate worker thread.

@kingluo I do think they are possible, we can create another fake request for them. thoughts?

[suikabreaker]

And about the API call arguments, how should I call function defined in the same module which calls the run_worker_thread

I can not understand you, can you give a simple sample?

-- a.lua
function _M.func()
-- ...
end
function _M.on_access()
    ngx.run_worker_thread("my_thread_pool", "a", "func")
end

@suikabreaker Have you tried the sample? Do you got an error or not?

it says I'm doing recursive module require.

[kingluo]

it says I'm doing recursive module require.

@suikabreaker No recursive in this case, because ngx.run_worker_thread is not defined in the loaded module.

[kingluo]

@suikabreaker Could you check the PR to see if it's satisfied?

[suikabreaker]

@suikabreaker Could you check the PR to see if it's satisfied?

I suppose shared memory and log APIs need fake request to be call? Shared memory matters in my case. But maybe I can find another way to do with. Anyway thanks for your patience and your PR.

[doujiang24]

1. string manipulate functions, including regex/encode/decode;
2. nginx enviroment, like ngx.get_phase() config.*
3. ngx.log()
4. shared memory ngx.shared

@suikabreaker I do think these are reasonable and possible.

[kingluo]

@suikabreaker Now it supports ngx.shared.DICT API.

About logging, it involves logging handler in nginx, which may cause race condition accessed from other threads.