nginx crash on nchan assertion

[isbear]

Hello.

We're using your build of nginx-mainline on our servers. Our automation tools try to keep the running process without restarts, so, on configuration changes we reload the service, and also automatic package updates for nginx are off. So, at the time of the crash we were running nginx-extras 1.21.1-1+ubuntu20.04.1+deb.sury.org+1 on recently-upgraded ubuntu 20.04. The process uptime was (according to systemd stats) 1 month and 5 days.

As I see it, the crash was probably triggered by certbot issuing a reload of nginx configuration, when trying to update certificate. The nginx error log is as follows:

2021/10/04 16:16:09 [info] 29838#29838: Using 131072KiB of shared memory for nchan in /etc/nginx/nginx.conf:23
2021/10/04 16:20:47 [notice] 38292#38292: signal process started
2021/10/04 16:20:55 [notice] 38517#38517: signal process started
ter process /usr/sbin/nginx -g daemon on; master_process on;: /build/nginx-Fe6XOS/nginx-1.21.1/debian/modules/nchan/src/store/memory/memstore.c:720: nchan_store_init_worker: Assertion `procslot_found == 1' failed.                                                                                                                               2021/10/04 16:21:05 [alert] 3967#3967: worker process 39372 exited on signal 6
2021/10/04 16:21:05 [alert] 3967#3967: shared memory zone "memstore" was locked by 39372
ter process /usr/sbin/nginx -g daemon on; master_process on;: /build/nginx-Fe6XOS/nginx-1.21.1/debian/modules/nchan/src/store/memory/memstore.c:720: nchan_store_init_worker: Assertion `procslot_found == 1' failed.                                                                                                                               2021/10/04 16:21:05 [alert] 3967#3967: worker process 39373 exited on signal 6
2021/10/04 16:21:05 [alert] 3967#3967: shared memory zone "memstore" was locked by 39373
ter process /usr/sbin/nginx -g daemon on; master_process on;: /build/nginx-Fe6XOS/nginx-1.21.1/debian/modules/nchan/src/store/memory/memstore.c:720: nchan_store_init_worker: Assertion `procslot_found == 1' failed.                                                                                                                               2021/10/04 16:21:05 [alert] 3967#3967: worker process 39374 exited on signal 6
2021/10/04 16:21:05 [alert] 3967#3967: shared memory zone "memstore" was locked by 39374
ter process /usr/sbin/nginx -g daemon on; master_process on;: /build/nginx-Fe6XOS/nginx-1.21.1/debian/modules/nchan/src/store/memory/memstore.c:720: nchan_store_init_worker: Assertion `procslot_found == 1' failed.                                                                                                                               
...

The log continues to repeat with different process IDs until manual service restart by admin.

Googling around I found the bug in nchan that is closed with resolution that it was caused by running mod-nchan, built for nginx patched with openresty, under vanilla nginx. But is that the case here? We're running nginx with both mod-lua and mod-nchan enabled.

To Reproduce

Unfortunately, I cannot provide instructions to reproduce this bug, and so far it only happened once (but we switched to your packages recently too).

Since this is an one-off case so far and may be a bug in the mod-nchan, this bugreport is mostly for information, feel free to close. If I will get more information on this I'll reopen.

Distribution (please complete the following information):

• OS: Ubuntu focal
• Architecture: amd64
• Repository: ppa:ondrej/nginx-mainline

Package(s) (please complete the following information): At the time there was installed package nginx-extras of version 1.21.1-1+ubuntu20.04.1+deb.sury.org+1, since then we've upgraded to 1.21.3-1+ubuntu20.04.1+deb.sury.org+1.

[oerdnj]

Yes, both stock Debian and DEB.SURY.ORG have openresty patch applied.

[yetzt]

i had the same symptoms with ubuntu 20.04, nginx 1.17.10 built from the ubuntu-sources with nchan 1.2.7.