uwsgi
uwsgi copied to clipboard
unbit
Segmentation fault during worker shutdown
I get occasional segfaults from uWSGI when a worker shuts down (for example, by reaching its max-requests setting). I haven't been able to reproduce the issue consistently so far, but the stacktrace gives a hint that openssl of version 1.1.0 might be involved (hence, may be related to #1395, and probably to #1495 and #1569). The issue is not critical (connections are not dropped for real users), but it is still annoying.
uWSGI in our case is managed by Supervisor, and runs a Django app.
Versions:
uWSGI 2.0.15
Django 1.9.13
Python 2.7.13
OpenSSL 1.1.0f
Debian 9.1
Linux 4.9.0-3-amd64
How I start uWSGI:
/home/user/app/current/venv/bin/uwsgi
--workers=16 --http 127.0.0.1:8200
--max-requests=999
--harakiri=180
--buffer-size=32768
--master --module app.wsgi:application
--py-call-osafterfork --enable-threads --single-interpreter
--disable-write-exception --ignore-sigpipe --ignore-write-errors
--lazy-apps --touch-chain-reload /home/user/run/uwsgi_chain
--hook-post-fork chdir:/home/user/app/current/src/
--http-timeout=180 --http-connect-timeout=180
Full stacktrace:
...The work of process 4637 is done. Seeya!
!!! uWSGI process 4637 got Segmentation Fault !!!
*** backtrace of 4637 ***
/home/user/app/current/venv/bin/uwsgi(uwsgi_backtrace+0x35) [0x55c132a5ae75]
/home/user/app/current/venv/bin/uwsgi(uwsgi_segfault+0x23) [0x55c132a5b233]
/lib/x86_64-linux-gnu/libc.so.6(+0x33030) [0x7f029422d030]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(ERR_put_error+0x23) [0x7f029546dd93]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(X509_load_cert_crl_file+0xf3) [0x7f02954e9883]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(+0x1c88fa) [0x7f02954e98fa]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(X509_STORE_load_locations+0x3f) [0x7f02954ec55f]
/home/user/app/current/venv/lib/python2.7/lib-dynload/_ssl.x86_64-linux-gnu.so(+0xd2d6) [0x7f028fd382d6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x8541) [0x7f0294854091]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x86c) [0x7f02949bb15c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x670d) [0x7f029485225d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x86c) [0x7f02949bb15c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x670d) [0x7f029485225d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x86c) [0x7f02949bb15c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x670d) [0x7f029485225d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x86c) [0x7f02949bb15c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x670d) [0x7f029485225d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x86c) [0x7f02949bb15c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x13e6ad) [0x7f029490f6ad]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f02948a7543]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x36f7) [0x7f029484f247]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x86c) [0x7f02949bb15c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x670d) [0x7f029485225d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x86c) [0x7f02949bb15c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x670d) [0x7f029485225d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x86c) [0x7f02949bb15c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x670d) [0x7f029485225d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6840) [0x7f0294852390]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6840) [0x7f0294852390]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6840) [0x7f0294852390]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6840) [0x7f0294852390]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6840) [0x7f0294852390]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6840) [0x7f0294852390]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6840) [0x7f0294852390]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6840) [0x7f0294852390]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x86c) [0x7f02949bb15c]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x13e5b0) [0x7f029490f5b0]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f02948a7543]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x193cbc) [0x7f0294964cbc]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f02948a7543]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7f02949ba587]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x5b5f2) [0x7f029482c5f2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) [0x7f0296149494]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f02942e2aff]
*** end of backtrace ***
Any input is welcome.
I am running into a very similar issue with Python 3, running New Relic with Flask's demo app Flaskr (https://github.com/pallets/flask/tree/0.12.2/examples/flaskr), both in a Docker container and directly on the host.
# Dockerfile
FROM debian:stretch
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -qy curl python3-pip
RUN curl -Lo /tmp/flask.tar.gz https://github.com/pallets/flask/archive/0.12.2.tar.gz \
&& mkdir /tmp/flask \
&& tar zxf /tmp/flask.tar.gz -C /tmp/flask --strip-components 1 \
&& mv /tmp/flask/examples/flaskr /root/flaskr \
&& rm -rf /tmp/flask /tmp/flask.tar.gz
WORKDIR /root/flaskr
ENV FLASK_APP=flaskr LC_ALL=C.UTF-8 LANG=C.UTF-8
RUN pip3 install flask==0.12.2 newrelic uwsgi==2.0.15 \
&& pip3 install -e . \
&& flask initdb
CMD newrelic-admin run-program uwsgi --master --disable-logging --enable-threads --single-interpreter --py-call-osafterfork --max-requests 999 --http :8000 --wsgi-file flaskr/flaskr.py --callable app
# docker-compose.yml
version: '3'
services:
app:
build: .
environment:
NEW_RELIC_LICENSE_KEY: secret
ports:
- 127.0.0.1:8000:8000
When uWSGI instances are stopped (either when the max number of requests is reached or when the server is killed), they randomly crash with a segmentation fault.
app_1 | !!! uWSGI process 10 got Segmentation Fault !!!
app_1 | *** backtrace of 10 ***
app_1 | uwsgi(uwsgi_backtrace+0x35) [0x563ec4262f15]
app_1 | uwsgi(uwsgi_segfault+0x23) [0x563ec42632d3]
app_1 | /lib/x86_64-linux-gnu/libc.so.6(+0x33030) [0x7f0ea5497030]
app_1 | /lib/x86_64-linux-gnu/libcrypto.so.1.1(ERR_put_error+0x23) [0x7f0ea3d1ad93]
app_1 | /lib/x86_64-linux-gnu/libssl.so.1.1(OPENSSL_init_ssl+0x57) [0x7f0ea408dbb7]
app_1 | /lib/x86_64-linux-gnu/libssl.so.1.1(SSL_CTX_new+0x1e) [0x7f0ea4090f4e]
app_1 | /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xeb9e) [0x7f0ea42dbb9e]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x8baf4) [0x7f0ea5ac6af4]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyCFunction_Call+0xe9) [0x7f0ea5aac279]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x8039) [0x7f0ea5bdd6a9]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23) [0x7f0ea5c9f073]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xf7665) [0x7f0ea5b32665]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f0ea5c0e9a7]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x8c685) [0x7f0ea5ac7685]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x909a5) [0x7f0ea5acb9a5]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f0ea5c0e9a7]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xf47) [0x7f0ea5bd65b7]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23) [0x7f0ea5c9f073]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xf7568) [0x7f0ea5b32568]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f0ea5c0e9a7]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_CallFunctionObjArgs+0x18f) [0x7f0ea5c0ebdf]
app_1 | /usr/local/lib/python3.5/dist-packages/newrelic/packages/wrapt/_wrappers.cpython-35m-x86_64-linux-gnu.so(+0x3d44) [0x7f0ea2401d44]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f0ea5c0e9a7]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x3bed) [0x7f0ea5bd925d]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f0ea5bdba4f]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f0ea5bdba4f]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x7fad) [0x7f0ea5bdd61d]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f0ea5bdba4f]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f0ea5bdba4f]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f0ea5bdba4f]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23) [0x7f0ea5c9f073]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xf7665) [0x7f0ea5b32665]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f0ea5c0e9a7]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xf47) [0x7f0ea5bd65b7]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23) [0x7f0ea5c9f073]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xf7665) [0x7f0ea5b32665]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f0ea5c0e9a7]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xf47) [0x7f0ea5bd65b7]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23) [0x7f0ea5c9f073]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xf7665) [0x7f0ea5b32665]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f0ea5c0e9a7]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xf47) [0x7f0ea5bd65b7]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f0ea5bdba4f]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f0ea5bdba4f]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f0ea5bdba4f]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x7fad) [0x7f0ea5bdd61d]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f0ea5bdba4f]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f0ea5bdba4f]
app_1 | /lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f0ea5c9ef94]
app_1 | *** end of backtrace ***
The problem still happens with option --skip-atexit or --skip-atexit-teardown.
So, in both cases it's calling out to libssl, but breaking in different places.
And with the reference to wrapt, am I guessing right you're using "requests" to access a HTTPS endpoint?
We are seeing the same issue. Another data point can't hurt:
/opt/xxx/bin/newrelic-admin run-program /opt/xxx/bin/uwsgi --ini /etc/xxx/pyramid.ini --enable-threads --thunder-lock
[uwsgi]
socket = 127.0.0.1:8000
home = /opt/xxx
paste = config:/etc/xxx/pyramid.ini
virtualenv = /opt/xxx
processes = 16
buffer-size = 64000
master = true
uid = xxx
gid = xxx
max-requests = 500
max-requests-delta = 100
http = 0.0.0.0:6543
wsgi-env-behavior = holy
We've tried with and without the wsgi-env-behavior = holy line - no change in segfaults either way.
We're using uwsgi 2.0.15 on Debian Stable with Python 3.5.3. Note we are using New Relic, but no other exotic configurations/options/code.
The traceback:
Jan 5 12:01:21 newrelic-admin[25501]: ...The work of process 19060 is done. Seeya!
Jan 5 12:01:21 newrelic-admin[25501]: 2018-01-05 12:01:21,940 (19060/b'uWSGIWorker9Core0') newrelic.core.agent INFO - New Relic Python Agent Shutdown
Jan 5 12:01:22 newrelic-admin[25501]: !!! uWSGI process 19060 got Segmentation Fault !!!
Jan 5 12:01:22 newrelic-admin[25501]: *** backtrace of 19060 ***
Jan 5 12:01:22 newrelic-admin[25501]: /opt/xxx/bin/uwsgi(uwsgi_backtrace+0x2a) [0x55888d2ae6ea]
Jan 5 12:01:22 newrelic-admin[25501]: /opt/xxx/bin/uwsgi(uwsgi_segfault+0x23) [0x55888d2aead3]
Jan 5 12:01:22 newrelic-admin[25501]: /lib/x86_64-linux-gnu/libc.so.6(+0x33030) [0x7f8e3641a030]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(ERR_put_error+0x23) [0x7f8e32089d93]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libssl.so.1.1(OPENSSL_init_ssl+0x57) [0x7f8e323fcbb7]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libssl.so.1.1(SSL_CTX_new+0x1e) [0x7f8e323fff4e]
Jan 5 12:01:22 newrelic-admin[25501]: /opt/crm.web/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xeb9e) [0x7f8e32655b9e]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x8baf4) [0x7f8e36a49af4]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyCFunction_Call+0xe9) [0x7f8e36a2f279]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x8039) [0x7f8e36b606a9]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23) [0x7f8e36c22073]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xf7665) [0x7f8e36ab5665]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f8e36b919a7]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x8c685) [0x7f8e36a4a685]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x909a5) [0x7f8e36a4e9a5]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f8e36b919a7]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xf47) [0x7f8e36b595b7]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23) [0x7f8e36c22073]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xf7568) [0x7f8e36ab5568]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f8e36b919a7]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_CallFunctionObjArgs+0x18f) [0x7f8e36b91bdf]
Jan 5 12:01:22 newrelic-admin[25501]: /opt/xxx/lib/python3.5/site-packages/newrelic/packages/wrapt/_wrappers.cpython-35m-x86_64-linux-gnu.so(+0x3bab) [0x7f8e304f4bab]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f8e36b919a7]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x3bed) [0x7f8e36b5c25d]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f8e36b5ea4f]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f8e36b5ea4f]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x7fad) [0x7f8e36b6061d]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f8e36b5ea4f]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f8e36b5ea4f]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f8e36b5ea4f]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23) [0x7f8e36c22073]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xf7665) [0x7f8e36ab5665]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f8e36b919a7]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xf47) [0x7f8e36b595b7]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23) [0x7f8e36c22073]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xf7665) [0x7f8e36ab5665]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f8e36b919a7]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xf47) [0x7f8e36b595b7]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23) [0x7f8e36c22073]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0xf7665) [0x7f8e36ab5665]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyObject_Call+0x47) [0x7f8e36b919a7]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xf47) [0x7f8e36b595b7]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f8e36b5ea4f]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f8e36b5ea4f]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f8e36b5ea4f]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x7fad) [0x7f8e36b6061d]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f8e36b5ea4f]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x63df) [0x7f8e36b5ea4f]
Jan 5 12:01:22 newrelic-admin[25501]: /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0(+0x263f94) [0x7f8e36c21f94]
Jan 5 12:01:22 newrelic-admin[25501]: *** end of backtrace ***
we have the same problem. I removed newrelic for the time being . maybe that is somehow installing an older lib SSL binary.
UPDATE = after removing the newrelic library and installing psycopg2 from source this problem went away.
Originally I thought this issue is related to psycopg2 installing binary libssl. https://github.com/psycopg/psycopg2/issues/543 but even after installing psycopg2 from source code we are still seeing seeing this error.
We are running our django app in this container:
FROM python:2.7.15-stretch
RUN apt-get update && \
apt-get install -y \
nginx \
openssl ca-certificates \
libpq-dev \
python-pip python-dev python-setuptools uwsgi-plugin-python \
sed \
trickle \
supervisor \
nano \
wget \
curl \
cron \
redis-tools \
zip \
unzip \
&& rm -rf /var/lib/apt/lists/*
[2018-07-10T00:26:34-07:00] !!! uWSGI process 39 got Segmentation Fault !!!
[2018-07-10T00:26:34-07:00] *** backtrace of 39 ***
[2018-07-10T00:26:34-07:00] /usr/local/bin/uwsgi(uwsgi_backtrace+0x35) [0x55bad5ea6295]
[2018-07-10T00:26:34-07:00] /usr/local/bin/uwsgi(uwsgi_segfault+0x23) [0x55bad5ea6643]
[2018-07-10T00:26:34-07:00] /lib/x86_64-linux-gnu/libc.so.6(+0x33060) [0x7f0d7de41060]
[2018-07-10T00:26:34-07:00] /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(ERR_put_error+0x23) [0x7f0d7ef0bd93]
[2018-07-10T00:26:34-07:00] /usr/lib/x86_64-linux-gnu/libssl.so.1.1(OPENSSL_init_ssl+0x57) [0x7f0d7f27ebb7]
[2018-07-10T00:26:34-07:00] /usr/lib/x86_64-linux-gnu/libssl.so.1.1(SSL_CTX_new+0x1e) [0x7f0d7f281f4e]
[2018-07-10T00:26:34-07:00] /usr/local/lib/python2.7/lib-dynload/_ssl.so(+0xb3ad) [0x7f0d7908a3ad]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(+0xc1089) [0x7f0d7e4a6089]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x669d) [0x7f0d7e4f1e1d]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(+0x85f7d) [0x7f0d7e46af7d]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f0d7e439ab3]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(+0xbc77d) [0x7f0d7e4a177d]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(+0xc11d5) [0x7f0d7e4a61d5]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f0d7e439ab3]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2a36) [0x7f0d7e4ee1b6]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(+0x85eac) [0x7f0d7e46aeac]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f0d7e439ab3]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyObject_CallFunctionObjArgs+0x177) [0x7f0d7e43a6a7]
[2018-07-10T00:26:34-07:00] /usr/local/lib/python2.7/site-packages/newrelic/packages/wrapt/_wrappers.so(+0x3b74) [0x7f0d76898b74]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f0d7e439ab3]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x54a7) [0x7f0d7e4f0c27]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ef6) [0x7f0d7e4f1676]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ef6) [0x7f0d7e4f1676]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6015) [0x7f0d7e4f1795]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6015) [0x7f0d7e4f1795]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ef6) [0x7f0d7e4f1676]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ef6) [0x7f0d7e4f1676]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(+0x85f7d) [0x7f0d7e46af7d]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f0d7e439ab3]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2a36) [0x7f0d7e4ee1b6]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(+0x85f7d) [0x7f0d7e46af7d]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f0d7e439ab3]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2a36) [0x7f0d7e4ee1b6]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(+0x85f7d) [0x7f0d7e46af7d]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f0d7e439ab3]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2a36) [0x7f0d7e4ee1b6]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ef6) [0x7f0d7e4f1676]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ef6) [0x7f0d7e4f1676]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ef6) [0x7f0d7e4f1676]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ef6) [0x7f0d7e4f1676]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ef6) [0x7f0d7e4f1676]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ef6) [0x7f0d7e4f1676]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(+0x85f7d) [0x7f0d7e46af7d]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7f0d7e439ab3]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2a36) [0x7f0d7e4ee1b6]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6015) [0x7f0d7e4f1795]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x6015) [0x7f0d7e4f1795]
[2018-07-10T00:26:34-07:00] /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7f0d7e4f54dc]
[2018-07-10T00:26:34-07:00] *** end of backtrace ***
I'm also getting the exact same as @ddehghan, around the same time at which newrelic cannot connect to its api.
logger.warning:
Data collector is not contactable. This can be because of a network issue or because of the data collector being restarted. In the event that contact cannot be made after a period of time then please report this problem to New Relic support for further investigation. The exception raised was ConnectionError(ProtocolError('Connection aborted.', gaierror(-3, 'Temporary failure in name resolution')),).
(Error above possibly due to AWS connectivity issues)
Just trying without newrelic to see if that resolves this for now.
edit:
it did stop the issue :)
FWIW. Often these crash on shutdown issues are caused by background daemon threads waking up while process exit is occurring and trying to do things after the Python interpreter has started destroying the interpreter, modules and module state. Usually it results in Python exceptions about accessing None objects, because everything in modules have been reset, but technically destruction of module state could result in cleaned up state in linked libraries as well, so if code then tried to use that library, it could blow up.
I have seen these sorts of issues occasionally with mod_wsgi as well, and newrelic would cause the None issue I mentioned. My only caveat on this for uwsgi is that I am not sure that uwsgi properly destroys the interpreter on process shutdown and just exits the process without destroying it. Vague recollection this may be the case as don't remember atexit callbacks ever working with it, although that may be the Python problem where atexit callbacks don't work in sub interpreters, which mod_wsgi works around and implements.
Seems I was on the right path, it being a bad interaction with something starting its own thread...
Hello all,
I was able to create a minimal reproduction of this segfault without the New Relic agent installed. It seems it has to do with ssl and not threads.
import atexit
try:
# PY3
from http.client import HTTPSConnection
except ImportError:
# PY2
from httplib import HTTPSConnection
def myatexit():
HTTPSConnection('https://example.com', port=443)
def application(environ, start_response):
status = '200 OK'
response_headers = [('Content-type', 'text/plain')]
start_response(status, response_headers)
return [b'*']
atexit.register(myatexit)
Using this Dockerfile. Note that I get a segfault on both Python 3.6.6 and Python 2.7.15.
FROM python:2.7.15
#FROM python:3.6.6
WORKDIR /data
RUN pip install uwsgi==2.0.17.1
CMD uwsgi \
--http :8000 \
--wsgi app:application \
--max-requests 3 \
--min-worker-lifetime 0 \
--master
ADD app.py /data
The reason New Relic keeps popping up is because the agent makes requests to the backend services during an atexit call using ssl.
!!! uWSGI process 10 got Segmentation Fault !!! [23/2650]
gateway "uWSGI http 1" has been buried (pid: 9)
*** backtrace of 10 ***
uwsgi(uwsgi_backtrace+0x35) [0x55e4dcb0d295]
uwsgi(uwsgi_segfault+0x23) [0x55e4dcb0d643]
/lib/x86_64-linux-gnu/libc.so.6(+0x33060) [0x7fa494065060]
/lib/x86_64-linux-gnu/libpthread.so.0(__pthread_rwlock_rdlock+0) [0x7fa496082f30]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(CRYPTO_THREAD_read_lock+0x9) [0x7fa4951a0ec9]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(+0x14c5d0) [0x7fa49512f5d0]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1(ERR_reason_error_string+0x59) [0x7fa49512fa69]
/usr/local/lib/python2.7/lib-dynload/_ssl.so(+0x83c2) [0x7fa4903013c2]
/usr/local/lib/python2.7/lib-dynload/_ssl.so(+0xb425) [0x7fa490304425]
/usr/local/lib/libpython2.7.so.1.0(+0xc1089) [0x7fa4946ca089]
/usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x669d) [0x7fa494715e1d]
/usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7fa4947194dc]
/usr/local/lib/libpython2.7.so.1.0(+0x85eac) [0x7fa49468eeac]
/usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fa49465dab3]
/usr/local/lib/libpython2.7.so.1.0(+0xbc77d) [0x7fa4946c577d]
/usr/local/lib/libpython2.7.so.1.0(+0xc11d5) [0x7fa4946ca1d5]
/usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fa49465dab3]
/usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x54a7) [0x7fa494714c27]
/usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7fa4947194dc]
/usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ef6) [0x7fa494715676]
/usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7fa4947194dc]
/usr/local/lib/libpython2.7.so.1.0(+0x85f7d) [0x7fa49468ef7d]
/usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fa49465dab3]
/usr/local/lib/libpython2.7.so.1.0(+0x65f7c) [0x7fa49466ef7c]
/usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fa49465dab3]
/usr/local/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fa49470f1a7]
/usr/local/lib/libpython2.7.so.1.0(PyInstance_New+0x6c) [0x7fa494676d5c]
/usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fa49465dab3]
/usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x54a7) [0x7fa494714c27]
/usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7fa4947194dc]
/usr/local/lib/libpython2.7.so.1.0(+0x85f7d) [0x7fa49468ef7d]
/usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fa49465dab3]
/usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2a36) [0x7fa4947121b6]
/usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8bc) [0x7fa4947194dc]
/usr/local/lib/libpython2.7.so.1.0(+0x85eac) [0x7fa49468eeac]
/usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fa49465dab3]
/usr/local/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fa49470f1a7]
/usr/local/lib/libpython2.7.so.1.0(Py_Finalize+0xb7) [0x7fa49473e7d7]
uwsgi(uwsgi_plugins_atexit+0x81) [0x55e4dcb0a301]
/lib/x86_64-linux-gnu/libc.so.6(+0x35940) [0x7fa494067940]
/lib/x86_64-linux-gnu/libc.so.6(+0x3599a) [0x7fa49406799a]
uwsgi(+0x3f29f) [0x55e4dcac229f]
uwsgi(end_me+0x25) [0x55e4dcb0a345]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0) [0x7fa4960880c0]
/lib/x86_64-linux-gnu/libc.so.6(epoll_wait+0x13) [0x7fa49411b0a3]
uwsgi(event_queue_wait+0x25) [0x55e4dcb00535]
uwsgi(wsgi_req_accept+0x13a) [0x55e4dcabfa2a]
uwsgi(simple_loop_run+0xb6) [0x55e4dcb090c6]
uwsgi(simple_loop+0x10) [0x55e4dcb08ef0]
uwsgi(uwsgi_ignition+0x1b3) [0x55e4dcb0d8b3]
uwsgi(uwsgi_worker_run+0x28d) [0x55e4dcb1227d]
uwsgi(+0x8f87c) [0x55e4dcb1287c]
uwsgi(+0x3bdbe) [0x55e4dcabedbe]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7fa4940522e1]
uwsgi(_start+0x2a) [0x55e4dcabedea]
*** end of backtrace ***
worker 1 buried after 1 seconds
goodbye to uWSGI.
Hi there,
Hope this issue is finally fixed in 2.0.18 since it includes #1879 fix.
Could anyone please try the new release to confirm if the issue could be closed?
Thank you!
Hello. After trying new release the issue still remains on python 3.7:
gevent==1.4.0 uWSGI==2.0.18
[uwsgi] chdir = /app http-socket = /tmp/{home_user}-websocket.sock master = true chmod-socket = 666 no-orphans = true die-on-term = true memory-report = true gevent = 1000 module = config.wsgi_websocket logto = /tmp/{home_user}-websocket-uwsgi.log threads = 2 processes = 1 http-websockets = true
DAMN ! worker 1 (pid: 109) died :( trying respawn ... Respawned uWSGI worker 1 (new pid: 110) *** running gevent loop engine [addr:0x5568ba853fd0] *** !!! uWSGI process 110 got Segmentation Fault !!! *** backtrace of 110 *** uwsgi(uwsgi_backtrace+0x35) [0x5568ba82ccf5] uwsgi(uwsgi_segfault+0x23) [0x5568ba82d0a3] /lib/x86_64-linux-gnu/libc.so.6(+0x33060) [0x7f6228d40060] uwsgi(wsgi_req_setup+0xa) [0x5568ba7df85a] uwsgi(py_uwsgi_gevent_main+0x4a) [0x5568ba853a9a] /usr/local/lib/libpython3.7m.so.1.0(PyCFunction_Call+0x9a) [0x7f622937e81a] /usr/local/lib/python3.7/site-packages/gevent/libev/corecext.cpython-37m-x86_64-linux-gnu.so(gevent_callback_io+0xa5) [0x7f62254b5335] /usr/local/lib/python3.7/site-packages/gevent/libev/corecext.cpython-37m-x86_64-linux-gnu.so(ev_invoke_pending+0x55) [0x7f62254880a5] /usr/local/lib/python3.7/site-packages/gevent/libev/corecext.cpython-37m-x86_64-linux-gnu.so(ev_run+0x4f1) [0x7f62254b36e1] /usr/local/lib/python3.7/site-packages/gevent/libev/corecext.cpython-37m-x86_64-linux-gnu.so(+0x41faa) [0x7f62254b3faa] /usr/local/lib/libpython3.7m.so.1.0(_PyMethodDef_RawFastCallKeywords+0x2f3) [0x7f622937cc73] /usr/local/lib/libpython3.7m.so.1.0(_PyMethodDescr_FastCallKeywords+0x49) [0x7f6229385329] /usr/local/lib/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x7cbb) [0x7f622935593b] /usr/local/lib/libpython3.7m.so.1.0(+0x68bd0) [0x7f622934cbd0] /usr/local/lib/libpython3.7m.so.1.0(_PyFunction_FastCallDict+0x2a6) [0x7f622937c6b6] /usr/local/lib/libpython3.7m.so.1.0(_PyObject_Call_Prepend+0xbd) [0x7f622937d78d] /usr/local/lib/libpython3.7m.so.1.0(PyObject_Call+0x65) [0x7f622937e9a5] /usr/local/lib/python3.7/site-packages/greenlet.cpython-37m-x86_64-linux-gnu.so(+0x3667) [0x7f6225060667] /usr/local/lib/python3.7/site-packages/greenlet.cpython-37m-x86_64-linux-gnu.so(+0x2ff6) [0x7f622505fff6] /usr/local/lib/python3.7/site-packages/greenlet.cpython-37m-x86_64-linux-gnu.so(+0x3f5b) [0x7f6225060f5b] /usr/local/lib/python3.7/site-packages/gevent/__greenlet_primitives.cpython-37m-x86_64-linux-gnu.so(+0x871f) [0x7f6224e5671f] /usr/local/lib/python3.7/site-packages/gevent/__greenlet_primitives.cpython-37m-x86_64-linux-gnu.so(+0x8dce) [0x7f6224e56dce] /usr/local/lib/python3.7/site-packages/gevent/__hub_local.cpython-37m-x86_64-linux-gnu.so(+0x7b9e) [0x7f622526bb9e] /usr/local/lib/python3.7/site-packages/gevent/_greenlet.cpython-37m-x86_64-linux-gnu.so(+0x138c9) [0x7f62247a28c9] /usr/local/lib/python3.7/site-packages/gevent/_greenlet.cpython-37m-x86_64-linux-gnu.so(+0x9166) [0x7f6224798166] /usr/local/lib/python3.7/site-packages/gevent/_greenlet.cpython-37m-x86_64-linux-gnu.so(+0x267b5) [0x7f62247b57b5] /usr/local/lib/python3.7/site-packages/gevent/_greenlet.cpython-37m-x86_64-linux-gnu.so(+0x154ab) [0x7f62247a44ab] /usr/local/lib/python3.7/site-packages/gevent/__hub_local.cpython-37m-x86_64-linux-gnu.so(+0x7b54) [0x7f622526bb54] /usr/local/lib/libpython3.7m.so.1.0(_PyObject_FastCallDict+0xbc) [0x7f622937d4ec] /usr/local/lib/libpython3.7m.so.1.0(_PyObject_Call_Prepend+0xbd) [0x7f622937d78d] /usr/local/lib/libpython3.7m.so.1.0(_PyObject_FastCallDict+0xbc) [0x7f622937d4ec] /usr/local/lib/libpython3.7m.so.1.0(PyObject_CallMethod+0x12e) [0x7f622937e72e] uwsgi(+0xb1534) [0x5568ba854534] uwsgi(uwsgi_ignition+0x135) [0x5568ba82d295] uwsgi(uwsgi_worker_run+0x275) [0x5568ba831c35] uwsgi(+0x8f22c) [0x5568ba83222c] uwsgi(+0x3c13e) [0x5568ba7df13e] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f6228d2d2e1] uwsgi(_start+0x2a) [0x5568ba7df16a] *** end of backtrace ***
Hi there @romabiker ! Looking at the stack trace posted, it appears possible that may be an unrelated crash since wsgi_req_setup called through py_uwsgi_gevent_main is involved in the stack trace. The problem described in this issue appears on worker shutdown while the segfault in the stack trace appears when handling requests if I'm interpreting that correctly.
@a-feld Thank you for your help and comment. The real reason of my Segmentation Fault turned out in misconfigured uwsgi ini file.
@ a-feld感谢您的帮助和评论。我的分段错误的真正原因是错误配置的uwsgi ini文件。 I have the same problem! Could you please tell me where the configuration problem is?
@romabiker - What was the misconfiguration in your uwsgi.ini that was causing the segfault?
uWSGI upgrade to 2.0.18 didn't resolve issue for me, but upgrade of psycopg2 from 2.7.3.2 to the latest 2.8.3 fixed things for me.
uWSGI upgrade to 2.0.18 didn't resolve issue for me, but upgrade of psycopg2 from 2.7.3.2 to the latest 2.8.3 fixed things for me.
It didnt work for me. @noonedeadpunk could you please let me know what python version are you using?
@a-feld Thank you for your help and comment. The real reason of my Segmentation Fault turned out in misconfigured uwsgi ini file.
@romabiker could you please share what you've found out?
I guess that it was resolved as pyscopg2 > 2.8 is now a source package so the libssl it is linking against is the very same of uwsgi.
@sylvioneto , @tianlilong , @kmans As I remember I've change threads and processes by workers. There's working config for me below:
[uwsgi]
chdir = /app
http-socket = /tmp/{home_user}-websocket.sock
master = true
chmod-socket = 666
gevent = 1000
module = config.wsgi_websocket
logto = /tmp/{home_user}-websocket-uwsgi.log
workers = 2
http-websockets = true
ignore-sigpipe = true
ignore-write-errors = true
disable-write-exception = true
uWSGI upgrade to 2.0.18 didn't resolve issue for me, but upgrade of psycopg2 from 2.7.3.2 to the latest 2.8.3 fixed things for me.
psycopg2-binary 2.8.3 did solve the problem on my side.
Perhaps this is relative to me using the python installation method with a non sentry-tested postgresql version (I'm using the postgresql-11.5 available with Debian 10)
...The work of process 11110 is done. Seeya!
!!! uWSGI process 11110 got Segmentation Fault !!!
*** backtrace of 11110 ***
/home/string/web/newsgallery/env/bin/uwsgi(uwsgi_backtrace+0x2c) [0x46bb0c]
/home/string/web/newsgallery/env/bin/uwsgi(uwsgi_segfault+0x21) [0x46bed1]
/lib/x86_64-linux-gnu/libc.so.6(+0x354b0) [0x7f1e02f7f4b0]
/home/string/web/newsgallery/env/lib/python3.6/site-packages/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so(+0xa5bd
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(+0xaeed7) [0x7f1e033c2ed7]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(_PyModule_ClearDict+0x22c) [0x7f1e0345fe0c]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(PyImport_Cleanup+0x42c) [0x7f1e035630bc]
/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0(Py_FinalizeEx+0x78) [0x7f1e03563398]
/home/string/web/newsgallery/env/bin/uwsgi(uwsgi_plugins_atexit+0x71) [0x468cb1]
/lib/x86_64-linux-gnu/libc.so.6(+0x39ff8) [0x7f1e02f83ff8]
/lib/x86_64-linux-gnu/libc.so.6(+0x3a045) [0x7f1e02f84045]
/home/string/web/newsgallery/env/bin/uwsgi() [0x421faf]
/home/string/web/newsgallery/env/bin/uwsgi() [0x46b526]
/home/string/web/newsgallery/env/bin/uwsgi() [0x46b558]
/home/string/web/newsgallery/env/bin/uwsgi(uwsgi_close_request+0x594) [0x422cd4]
/home/string/web/newsgallery/env/bin/uwsgi(simple_loop_run+0xdd) [0x467b3d]
/home/string/web/newsgallery/env/bin/uwsgi(simple_loop+0xe) [0x46791e]
/home/string/web/newsgallery/env/bin/uwsgi(uwsgi_ignition+0x22a) [0x46c1ba]
/home/string/web/newsgallery/env/bin/uwsgi(uwsgi_worker_run+0x27d) [0x4708fd]
/home/string/web/newsgallery/env/bin/uwsgi(uwsgi_init_worker_mount_apps+0) [0x470ee0]
/home/string/web/newsgallery/env/bin/uwsgi() [0x41ea2e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f1e02f6a830]
/home/string/web/newsgallery/env/bin/uwsgi(_start+0x29) [0x41ea59]
*** end of backtrace ***
Main process exited, code=exited, status=1/FAILURE
Getting segfault on worker shutdown but due to google/protobuf/pyext
As well am getting the same segfaults with google/protobuf/pyext, as @string-areeb
I am also getting issue on python3.6
/opt/virtualenv/<app_name>/bin/uwsgi(uwsgi_backtrace+0x2e) [0x482b1e] /opt/virtualenv/<app_name>/bin/uwsgi(uwsgi_segfault+0x21) [0x482eb1] /lib64/libc.so.6(+0x36340) [0x7f4a2911e340] /opt/virtualenv/<app_name>/lib/python3.6/site-packages/google/protobuf/pyext/_message.cpython-36m-x86_64-linux-gnu.so(+0xa5be0) [0x7f4a1584fbe0] /opt/virtualenv/<app_name>/bin/uwsgi() [0x50447f] /opt/virtualenv/<app_name>/bin/uwsgi() [0x505819] /opt/virtualenv/<app_name>/bin/uwsgi() [0x5ecee2] /opt/virtualenv/<app_name>/bin/uwsgi(_PyGC_CollectNoFail+0x31) [0x5edbd1] /opt/virtualenv/<app_name>/bin/uwsgi(PyImport_Cleanup+0x344) [0x5b6704] /opt/virtualenv/<app_name>/bin/uwsgi() [0x5c9b18] /opt/virtualenv/<app_name>/bin/uwsgi(uwsgi_plugins_atexit+0x71) [0x47fd41] /lib64/libc.so.6(+0x39c29) [0x7f4a29121c29] /lib64/libc.so.6(+0x39c77) [0x7f4a29121c77] /opt/virtualenv/<app_name>/bin/uwsgi() [0x43a3bf] /opt/virtualenv/<app_name>/bin/uwsgi(end_me+0x25) [0x47fd95] /lib64/libpthread.so.0(+0xf5d0) [0x7f4a2acba5d0] /opt/virtualenv/<app_name>/bin/uwsgi(end_me+0) [0x47fd70] /lib64/libpthread.so.0(+0xf5d0) [0x7f4a2acba5d0] /lib64/libpthread.so.0(+0x5b76) [0x7f4a2acb0b76] /opt/virtualenv/<app_name>/bin/uwsgi(uwsgi_lock_fast+0xd) [0x46e32d] /opt/virtualenv/<app_name>/bin/uwsgi(wsgi_req_accept+0x25d) [0x437e7d] /opt/virtualenv/<app_name>/bin/uwsgi(simple_loop_run+0xb6) [0x47eda6] /opt/virtualenv/<app_name>/bin/uwsgi(simple_loop+0xe) [0x47ebde] /opt/virtualenv/<app_name>/bin/uwsgi(uwsgi_ignition+0x192) [0x483102] /opt/virtualenv/<app_name>/bin/uwsgi(uwsgi_worker_run+0x2ed) [0x48794d] /opt/virtualenv/<app_name>/bin/uwsgi() [0x487f0f] /opt/virtualenv/<app_name>/bin/uwsgi(_start+0) [0x4371be] /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f4a2910a495] /opt/virtualenv/<app_name>/bin/uwsgi() [0x4371e7] *** end of backtrace ***
I am getting similar issue, I am using google vision api in flask application.
2021-02-08 18:18:34,073 GoogleVisionOCR DEBUG start to invoke google vision api 2021-02-08 18:18:42,898 GoogleVisionOCR DEBUG Received response 2021-02-08 18:18:42,900 ImagePredict DEBUG image_id: id1, result length: 95 [pid: 675|app: 0|req: 1/1] 172.17.0.1 () {44 vars in 619 bytes} [Mon Feb 8 18:18:33 2021] POST /v1/predict => generated 62576 bytes in 8943 msecs (HTTP/1.1 200) 3 headers in 114 bytes (179 switches on core 0) !!! uWSGI process 650 got Segmentation Fault !!! *** backtrace of 650 *** /usr/local/bin/uwsgi(uwsgi_backtrace+0x2a) [0x561c7f61187a] /usr/local/bin/uwsgi(uwsgi_segfault+0x23) [0x561c7f611c33] /lib/x86_64-linux-gnu/libc.so.6(+0x37840) [0x7f250eb56840] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2b19d5) [0x7f250c20c9d5] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2a8a90) [0x7f250c203a90] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2a98e9) [0x7f250c2048e9] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x18e4fd) [0x7f250c0e94fd] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2ae244) [0x7f250c209244] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2c5009) [0x7f250c220009] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x29df28) [0x7f250c1f8f28] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f250f4e9fa3] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f250ec184cf] *** end of backtrace *** DAMN ! worker 2 (pid: 650) died, killed by signal 11 :( trying respawn ... Respawned uWSGI worker 2 (new pid: 782) !!! uWSGI process 641 got Segmentation Fault !!! *** backtrace of 641 *** /usr/local/bin/uwsgi(uwsgi_backtrace+0x2a) [0x561c7f61187a] /usr/local/bin/uwsgi(uwsgi_segfault+0x23) [0x561c7f611c33] /lib/x86_64-linux-gnu/libc.so.6(+0x37840) [0x7f250eb56840] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2b19d5) [0x7f250c20c9d5] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2a8a90) [0x7f250c203a90] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2a98e9) [0x7f250c2048e9] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x18e4fd) [0x7f250c0e94fd] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2ae244) [0x7f250c209244] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2c5009) [0x7f250c220009] /usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x29df28) [0x7f250c1f8f28] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f250f4e9fa3] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f250ec184cf] *** end of backtrace *** DAMN ! worker 1 (pid: 641) died, killed by signal 11 :( trying respawn ... Respawned uWSGI worker 1 (new pid: 798)
This problem was related to grpc. resolved in this thread https://github.com/grpc/grpc/issues/23796