uWSGI segmentation fault when reaching lifetime limit after upgrade scipy to 1.9.1

[LimingXuIHME]

System info:

• uWSGI 2.0.20
• Python 3.8.12
• scipy 1.9.1
• OS: Debian GNU/Linux 10 (buster)
• K8s; docker

uWSGI config:

uwsgi_ini.pdf

Issue:

We run a python flask server using uWSGI on K8S. When using scipy 1.8.1, everything was fine.

Recently, we upgrade scipy to 1.9.0/1. Since then, when uWSGI workers reach the max max-worker-lifetime, we see the follow in the server console: !!! uWSGI process 33 got Segmentation Fault !!! and the flask server stops responding.

By running "ps -ef", we see the uWSGI process is still alive; thus, there is no core dump.

So, it seems to me the uWSGI workers fail to kill themselves.

Any suggestions?

Thanks, Liming

[daphtdazz]

We also see this (determined that it was an interaction between scipy and uwsgi by tedious process of removing things until it stopped).

It happens when we terminate uwsgi normally and when uwsgi terminates a worker at the end of its life. The problem for us was this was producing lots of core files in production and causing disk pressure on our k8s cluster.

We're working around this now and realistically the fix is going to be switch to a maintained asgi implementation rather than keep using uwsgi (although in fairness this is probably a bug in scipy not tidying up properly rather than uwsgi).

The crash we see in production just has the error message:

corrupted double-linked list

And the stack trace that I see on my Mac when I reproduce it looks like this:

!!! uWSGI process 60174 got Segmentation Fault !!!
*** backtrace of 60174 ***
0   uwsgi                               0x000000010d7737ab uwsgi_backtrace + 43
1   uwsgi                               0x000000010d773ce1 uwsgi_segfault + 49
2   libsystem_platform.dylib            0x00007ff818968e9d _sigtramp + 29
3   Python                              0x000000010ec3b558 _PyRuntime + 166384
4   Python                              0x000000010e8f18aa module_traverse + 93
5   Python                              0x000000010e9ef69d gc_collect_main + 269
6   Python                              0x000000010e9ef444 gc_collect_with_callback + 68
7   Python                              0x000000010e9f0826 PyGC_Collect + 100
8   Python                              0x000000010e9cdbb3 Py_FinalizeEx + 580
9   uwsgi                               0x000000010d773ca2 uwsgi_plugins_atexit + 162
10  libsystem_c.dylib                   0x00007ff8188008a5 __cxa_finalize_ranges + 409
11  libsystem_c.dylib                   0x00007ff8188006bf exit + 35
12  uwsgi                               0x000000010d723a86 uwsgi_exit + 22
13  uwsgi                               0x000000010d772bb8 end_me + 56
14  libsystem_platform.dylib            0x00007ff818968e9d _sigtramp + 29
15  ???                                 0x00007ff7b27dcc60 0x0 + 140701828238432
16  uwsgi                               0x000000010d725ccc wsgi_req_accept + 284
17  uwsgi                               0x000000010d77176c simple_loop_run + 380
18  uwsgi                               0x000000010d7714fe simple_loop + 142
19  uwsgi                               0x000000010d77882a uwsgi_ignition + 490
20  uwsgi                               0x000000010d778505 uwsgi_worker_run + 661
21  uwsgi                               0x000000010d7761c5 uwsgi_run + 1093
22  uwsgi                               0x000000010d773e8e main + 14
23  dyld                                0x00007ff8185a62cd start + 1805