use lua-nginx-module for nginx:1.16.1

[coeus-lei]

I dont know what happened ,if i use lua-nginx-module on nginx:1.16.1 verison when i start nginx:1.16.1
2020/05/30 05:21:14 [alert] 73242#0: worker process 74783 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74784 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74785 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74786 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74788 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74787 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74790 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74789 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74791 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74792 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74793 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74794 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74797 exited on signal 11 2020/05/30 05:21:14 [alert] 73242#0: worker process 74795 exited on signal 11 there are so many error log

[rainingmaster]

@coeus-lei Hi, it seems some problem with your Nginx worker during their running, they should be core dump and exit, so you can notice that the worker id are grow continually.

Maybe you can set your nginx.conf like this

worker_rlimit_core 10000m;
working_directory /usr/local/nginx/logs;

If worker core dump again, you can get your nginx's core dump file in /usr/local/nginx/logs.

[coeus-lei]

Thanks , I will try it again as you said ! laster I will report again

[theyosh]

Hi, the same thing here. And I found out that the functions:

https://github.com/openresty/lua-nginx-module/#init_by_lua and https://github.com/openresty/lua-nginx-module/#init_worker_by_lua etc are causing this.

I also have a init function which loads mysql databases. When I remove all the lua init functions, NGINX will start. But of course, I do not have mysql connections. So it is not a solution for me.

But I am afraid that with NGINX 1.16 and these 'init_by_lua..' functions are not working together.

Also testing with the example at: https://github.com/openresty/lua-nginx-module/#init_worker_by_lua will not work, and crash my NGINX during startup.

Also the options for @rainingmaster do not change this for me. So that is not working. Also I am not able to debug it. I do see in my logs also a log of these lines. They are all the same:

[63446.093862] nginx[22443]: segfault at 10 ip 00005555555b3d35 sp 00007fffffffdc30 error 4 in nginx[555555554000+102000]
[63446.093870] Code: ff 48 89 43 50 48 89 43 58 48 89 43 78 48 89 43 60 48 89 83 80 00 00 00 48 89 83 88 00 00 00 48 89 83 90 00 00 00 48 8b 45 28 <48> 8b 50 10 48 8b 80 f8 00 00 00 48 89 53 30 48 89 43 38 48 83 c4

nginx -V:

nginx version: nginx/1.16.1
built with OpenSSL 1.1.1  11 Sep 2018
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-hIpOlr/nginx-1.16.1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-mail=dynamic --with-mail_ssl_module --add-dynamic-module=/build/nginx-hIpOlr/nginx-1.16.1/debian/modules/http-auth-pam --add-dynamic-module=/build/nginx-hIpOlr/nginx-1.16.1/debian/modules/http-dav-ext --add-dynamic-module=/build/nginx-hIpOlr/nginx-1.16.1/debian/modules/http-echo --add-dynamic-module=/build/nginx-hIpOlr/nginx-1.16.1/debian/modules/http-upstream-fair --add-dynamic-module=/build/nginx-hIpOlr/nginx-1.16.1/debian/modules/http-subs-filter

For now, the only solution I have, is not upgrading to NGINX 1.16, which is doable now, as Ubuntu 18.04 does provide 1.14 which does work. But when 1.16 is becoming the next main version, that will crash a lot of NGINX setups...

[rainingmaster]

@theyosh , Hi

I also have a init function which loads mysql databases. When I remove all the lua init functions, NGINX will start. But of course, I do not have mysql connections. So it is not a solution for me.

Have you connect your mysql in ngx.timer.*, because ngx.socket.tcp can not work in both init_by_lua and init_worker_by_lua. And I think you can connect mysql in each request with pool instead of pre-connect?

So that is not working. Also I am not able to debug it. I do see in my logs also a log of these lines. They are all the same:

Not working means you can not get the core dump file under the path of working_directory ? Have you get the error log like 2020/05/30 05:21:14 [alert] 73242#0: worker process 74795 exited on signal 11? And does nginx have the permission to write in working_directory.

If problem still exists after your upgrade, maybe you can install your Openresty by official repo: https://openresty.org/linux-packages.html

[bjoe2k4]

Not sure if it helps, but i've been running nginx 1.19.0 since a few days and got no crash so far, also using init_by_lua to load some globals and init_worker_by_lua to start some background tasks via ngx.timer.every

nginx version: nginx/1.19.0
built by gcc 10.1.0 (GCC)
built with OpenSSL 1.1.1g  21 Apr 2020
TLS SNI support enabled
configure arguments: --prefix=/usr/local --with-ld-opt=-Wl,-rpath,/usr/local-resty/lib --conf-path=/etc/nginx/nginx.conf --error-log-path=/app/log/nginx.err --http-log-path=/app/log/nginx.acc --lock-path=/run/nginx/nginx.lock --pid-path=/run/nginx/nginx.pid --with-file-aio --with-http_gzip_static_module --with-http_stub_status_module --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-pcre --with-pcre-jit --without-http_scgi_module --without-http_split_clients_module --without-http_ssi_module --without-http_userid_module --without-http_uwsgi_module --without-http_upstream_ip_hash_module --without-http_browser_module --without-http_memcached_module --without-http_geo_module --add-module=modules/ngx_devel_kit --add-module=modules/lua-nginx-module --add-module=modules/headers-more-nginx-module --add-module=modules/echo-nginx-module --add-module=modules/ngx_brotli --add-module=modules/ngx_http_geoip2_module

[coeus-lei]

I already tested nginx 1.19 but it's not work ,i will try another way! thanks for everyone

[theyosh]

Ok, I removed all database code in my timer function. Still it does not work. So let summerise what I have done, and I am afraid, that is a bug in either the nginx lua module, or nginx, or lua... So here we go:

On my test laptop I run: nginx -V:

nginx version: nginx/1.16.1
built with OpenSSL 1.1.1  11 Sep 2018
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-hIpOlr/nginx-1.16.1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --with-mail=dynamic --with-mail_ssl_module --add-dynamic-module=/build/nginx-hIpOlr/nginx-1.16.1/debian/modules/http-auth-pam --add-dynamic-module=/build/nginx-hIpOlr/nginx-1.16.1/debian/modules/http-dav-ext --add-dynamic-module=/build/nginx-hIpOlr/nginx-1.16.1/debian/modules/http-echo --add-dynamic-module=/build/nginx-hIpOlr/nginx-1.16.1/debian/modules/http-upstream-fair --add-dynamic-module=/build/nginx-hIpOlr/nginx-1.16.1/debian/modules/http-subs-filter

This is installed from the PPA: https://launchpad.net/~nginx/+archive/ubuntu/stable Module libnginx-mod-http-lua

Package: libnginx-mod-http-lua
Version: 1.16.1-0+bionic1
Priority: optional
Section: httpd
Source: nginx
Maintainer: Debian Nginx Maintainers <[email protected]>
Installed-Size: 460 kB
Depends: libnginx-mod-http-ndk (= 1.16.1-0+bionic1), nginx-common (= 1.16.1-0+bionic1), libc6 (>= 2.14), libluajit-5.1-2 (>= 2.0.4+dfsg)
Download-Size: 173 kB
APT-Manual-Installed: yes
APT-Sources: http://ppa.launchpad.net/nginx/stable/ubuntu bionic/main amd64 Packages
Description: Lua module for Nginx
 Embed Lua runtime into nginx.
 .
 This module embeds Lua, via the standard Lua 5.1 interpreter or LuaJIT
 2.0/2.1, into Nginx and by leveraging Nginx's subrequests, allows the
 integration of the powerful Lua threads (Lua coroutines) into the Nginx event
 model.

I make the following nginx vhost config:

-- source: https://github.com/openresty/lua-nginx-module/#init_worker_by_lua
init_worker_by_lua '
     local delay = 3  -- in seconds
     local new_timer = ngx.timer.at
     local log = ngx.log
     local ERR = ngx.ERR
     local check

     check = function(premature)
         if not premature then
             -- do the health check or other routine work
             local ok, err = new_timer(delay, check)
             if not ok then
                 log(ERR, "failed to create timer: ", err)
                 return
             end
         end
     end

     local hdl, err = new_timer(delay, check)
     if not hdl then
         log(ERR, "failed to create timer: ", err)
         return
     end
 ';

server {
    listen 80 default_server;
    listen [::]:80 default_server;

	charset utf-8;
        error_page 500 502 503 504  /50x.html;
	root /opt/webdata/livecounter.theyosh.nl/wwwroot;
	server_name localhost;
	access_log   /var/log/nginx/livecounter.theyosh.nl.access.log;
	error_log /var/log/nginx/livecounter.theyosh.nl.error.log;

	location = / {
		expires 14d;
		# First attempt to serve request as file, then
		# as directory, then fall back to index.html
		try_files $uri $uri/ /index.html;
	}
}

And this is causing the core dumps. So, when an example of the module developer is not working, I have to conclude it is a bug that is not caused by my code! Sorry, but how do I have to see this? I am not looking for pointing to someone, but here is something wrong that is not caused by my software.

So, downgrading to NGINX 1.14 and lua mod.

nginx version: nginx/1.14.0 (Ubuntu)
built with OpenSSL 1.1.1  11 Sep 2018
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-GkiujU/nginx-1.14.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module

Mod lua:

Package: libnginx-mod-http-lua
Version: 1.14.0-0ubuntu1.7
Priority: optional
Section: universe/httpd
Source: nginx
Origin: Ubuntu
Maintainer: Ubuntu Developers <[email protected]>
Original-Maintainer: Debian Nginx Maintainers <[email protected]>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 467 kB
Depends: libnginx-mod-http-ndk (= 1.14.0-0ubuntu1.7), nginx-common (= 1.14.0-0ubuntu1.7), libc6 (>= 2.14), libluajit-5.1-2 (>= 2.0.4+dfsg)
Homepage: http://nginx.net
Download-Size: 150 kB
APT-Manual-Installed: yes
APT-Sources: ftp://ftpserv.tudelft.nl/pub/Linux/archive.ubuntu.com bionic-updates/universe amd64 Packages
Description: Lua module for Nginx
 Embed Lua runtime into nginx.
 .
 This module embeds Lua, via the standard Lua 5.1 interpreter or LuaJIT
 2.0/2.1, into Nginx and by leveraging Nginx's subrequests, allows the
 integration of the powerful Lua threads (Lua coroutines) into the Nginx event
 model.

And this IS working. So your 'init_worker_by_*' is broken from version 16.1 and onwards! So, therefore this means that more setups will be broken when NGINX is upgraded to version 16.1 or higher. As version 1.17 is also crashing, which is the next release in Ubuntu 20.04. So that means, an LTS update will break my nginx setup. So, how do we fix this?

Also there are NO dumps written to disk by NGINX even when the directory is owed by NGINX and world writeable...

[4xoc]

I can confirm that on Ubuntu 20.04 (focal) Nginx is still crashing when using init_worker_by_lua_block, even when the block is empty. Any progress on this? Any more details needed?

[Triple-Z]

@theyosh I get the same result in nginx v1.16.0. However the lua-nginx-module v0.10.17 claims that it is compatible with nginx v1.17.x, why the 1.17 is also crashing?

[knyar]

Note that Ubuntu seems to have made a decision to remove the libnginx-mod-http-lua package instead of fixing it. Patches that I submitted four months ago with fixes for hirsuite and focal have not been reviewed yet.

More details in https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1893753.

[theyosh]

Ok, I think I will switch to NGINX JavaScript module. https://nginx.org/en/docs/njs/ Using the NGINX repository: https://nginx.org/en/linux_packages.html#Ubuntu

[knyar]

FYI, there's been a few updates in the ubuntu bug report and there is a fixed package for Ubuntu focal published in the PPA.

[utkarsh2102]

Hiya,

For the Ubuntu bug for Focal, can you test the following PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4678/+packages

And let us know in the bug: https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1893753 if that worked for you? TIA! \o/