mod_wsgi icon indicating copy to clipboard operation
mod_wsgi copied to clipboard
verified publisher icon GrahamDumpleton

Importing GeoPandas or PyProj causes Apache/mod_wsgi crash

Open SteffenKr opened this issue 2 years ago • 5 comments

Hi, I am trying to create a website using Dash Leaflet, and I encountered an issue where importing GeoPandas or PyProj libraries causes the site to crash when running with mod_wsgi. I have isolated the problem by using a minimal example, and as soon as I include the import statement for either library, the site fails to load. In the Apache log, I see the following error messages being repeated in a loop:

[Mon May 22 13:30:16.863554 2023] [wsgi:info] [pid 146890:tid 140393768855424] mod_wsgi (pid=146890): Starting process 'test-proj' with uid=33, gid=33 and threads=15. [Mon May 22 13:30:16.865497 2023] [wsgi:info] [pid 146890:tid 140393768855424] mod_wsgi (pid=146890): Python home /var/www/test-proj/venv2. [Mon May 22 13:30:16.865557 2023] [wsgi:info] [pid 146890:tid 140393768855424] mod_wsgi (pid=146890): Initializing Python. [Mon May 22 13:30:16.877459 2023] [wsgi:info] [pid 146890:tid 140393768855424] mod_wsgi (pid=146890): Attach interpreter ''. [Mon May 22 13:30:16.884055 2023] [wsgi:info] [pid 146890:tid 140393768855424] mod_wsgi (pid=146890): Adding '/var/www/test-proj' to path. [Mon May 22 13:30:16.887385 2023] [wsgi:info] [pid 146890:tid 140393768855424] mod_wsgi (pid=146890): Imported 'mod_wsgi'. [Mon May 22 13:30:16.887440 2023] [wsgi:info] [pid 146890:tid 140393768855424] mod_wsgi (pid=146890, process='test-proj', application=''): Loading Python script file '/var/www/test-proj/test-proj.wsgi'. [Mon May 22 13:30:17.864567 2023] [core:notice] [pid 896:tid 140393768855424] AH00051: child pid 146890 exit signal Abort (6), possible coredump in /etc/apache2 [Mon May 22 13:30:17.864604 2023] [wsgi:info] [pid 896:tid 140393768855424] mod_wsgi (pid=146890): Process 'test-proj' has died, deregister and restart it. [Mon May 22 13:30:17.864613 2023] [wsgi:info] [pid 896:tid 140393768855424] mod_wsgi (pid=146890): Process 'test-proj' terminated by signal 6 [Mon May 22 13:30:17.864619 2023] [wsgi:info] [pid 896:tid 140393768855424] mod_wsgi (pid=146890): Process 'test-proj' has been deregistered and will no longer be monitored. [Mon May 22 13:30:17.866740 2023] [wsgi:info] [pid 146896:tid 140393768855424] mod_wsgi (pid=146896): Starting process 'test-proj' with uid=33, gid=33 and threads=15.

The site loads successfully when I remove the import statement for GeoPandas or PyProj. I have also tested this issue on other running projects, and they also crash.

Here are some additional details:

conf file:

LoadModule wsgi_module "/var/www/test-proj/venv2/lib64/python3.10/site-packages/mod_wsgi/server/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so"
WSGIDaemonProcess test-proj python-home=/var/www/test-proj/venv2 python-path=/var/www/test-proj display-name=%{GROUP}
WSGIScriptAlias /test-proj /var/www/test-proj/test-proj.wsgi process-group=test-proj application-group=%{GLOBAL}

<Directory /var/www/test-proj>
    <Files test-proj.wsgi>
		Require all granted
	</Files>
</Directory>

wsgi file:

import sys
sys.path.insert(0, "/var/www/test-proj/")
from index_test_2 import app
application = app.server

example code (index_test_2.py):

import dash
import dash_leaflet as dl

# import geopandas

app = dash.Dash(__name__, requests_pathname_prefix='/test-proj/')

app.layout = dl.Map(dl.TileLayer(), style={'height': '100vh'})

if __name__ == '__main__':
    app.run_server(port=8050, debug=True)

Python version in the virtual environment (venv2):

Python 3.10.6

Packages installed in venv2 (pip freeze):

attrs==23.1.0 blinker==1.6.2 certifi==2023.5.7 click==8.1.3 click-plugins==1.1.1 cligj==0.7.2 dash==2.9.3 dash-core-components==2.0.0 dash-html-components==2.0.0 dash-leaflet==0.1.23 dash-table==5.0.0 Fiona==1.9.4 Flask==2.3.2 geobuf==1.1.1 geopandas==0.13.0 itsdangerous==2.1.2 Jinja2==3.1.2 MarkupSafe==2.1.2 mod-wsgi==4.9.4 numpy==1.24.3 packaging==23.1 pandas==2.0.1 plotly==5.14.1 protobuf==4.23.0 pyproj==3.5.0 python-dateutil==2.8.2 pytz==2023.3 shapely==2.0.1 six==1.16.0 tenacity==8.2.2 tzdata==2023.3 Werkzeug==2.3.4

Apache version (apache2 -v):

Server version: Apache/2.4.52 (Ubuntu) Server built: 2023-03-08T17:32:01

OS (lsb_release -a):

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.2 LTS Release: 22.04 Codename: jammy

I would appreciate any help in resolving this issue. Thank you!

SteffenKr avatar May 22 '23 11:05 SteffenKr

Anything based on numpy will not work in Python sub interpreters. You need to force the use of the Python main interpreter. See:

  • https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api

about setting WSGIApplicationGroup directive.

GrahamDumpleton avatar May 22 '23 12:05 GrahamDumpleton

Thanks for the fast answer. Doesn't this line perform that function?

WSGIScriptAlias /test-proj /var/www/test-proj/test-proj.wsgi process-group=test-proj application-group=%{GLOBAL}

When I remove the last part of that line and add another line with "WSGIApplicationGroup %{GLOBAL}", the only difference is, that i get an Internal Server error and this in the logs:

Truncated or oversized response headers received from daemon process

Additionally, I have no issues with importing pandas or scipy either way.

SteffenKr avatar May 22 '23 13:05 SteffenKr

Ahh yes, sorry, was in a rush to give a response just as was going to sleep. The application-group should do the same. I didn't read properly and assumed were not using it since application-group isn't generally mentioned prominently in docs.

So issue is that the daemon process crashes. This is generally due to same issue though so is strange. Given no obvious problem with configuration, are you able to try and extract a C stacktrace for why the crash occurs. See:

  • https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#debugging-crashes-with-gdb

GrahamDumpleton avatar May 22 '23 20:05 GrahamDumpleton

No problem, I am just glad to get help. Before I opened this issue i searched in a few others with the hope I could find a fix. In one of your answeres I found application-group.

This is the .conf right now:

LoadModule wsgi_module "/var/www/test-proj/venv2/lib64/python3.10/site-packages/mod_wsgi/server/mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so"
WSGIDaemonProcess test-proj python-home=/var/www/test-proj/venv2 python-path=/var/www/test-proj display-name=%{GROUP} threads=1
WSGIScriptAlias /test-proj /var/www/test-proj/test-proj.wsgi process-group=test-proj
WSGIApplicationGroup %{GLOBAL}

<Directory /var/www/test-proj>
    <Files test-proj.wsgi>
		Require all granted
	</Files>
</Directory>

I then took the pid from the apache log file and attached gdb with sudo gdb /usr/sbin/apache2 <pid> (read that, since I use ubuntu, I had to deviate from the documentation). then i typed cont and thread apply all bt and refreshed the page. I am new to this, but I guess, Thread 4 is the relevant one. And I just imported geopandas, but pyproj gets mentioned. I checked and its a dependency of geopandas. Here is what I got:

(gdb) cont
Continuing.
thread apply all bt
[New Thread 0x7f0efc9ff640 (LWP 223085)]
[New Thread 0x7f0ef6185640 (LWP 223086)]
[New Thread 0x7f0ef5984640 (LWP 223087)]

Thread 4 "apache2" received signal SIGABRT, Aborted.
[Switching to Thread 0x7f0efe194640 (LWP 222995)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=139702369338944) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) thread apply all bt

Thread 7 (Thread 0x7f0ef5984640 (LWP 223087) "apache2"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f0ef7f55be0 <thread_status+352>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f0ef7f55be0 <thread_status+352>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f0ef7f55be0 <thread_status+352>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f0f09e2fac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f0ef7f55b90 <thread_status+272>, cond=0x7f0ef7f55bb8 <thread_status+312>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7f0ef7f55bb8 <thread_status+312>, mutex=0x7f0ef7f55b90 <thread_status+272>) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f0ef64bd43b in blas_thread_server () from target:/var/www/test-proj/venv2/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-15028c96.3.21.so
#6  0x00007f0f09e30b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007f0f09ec2a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7f0ef6185640 (LWP 223086) "apache2"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f0ef7f55b60 <thread_status+224>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f0ef7f55b60 <thread_status+224>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f0ef7f55b60 <thread_status+224>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f0f09e2fac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f0ef7f55b10 <thread_status+144>, cond=0x7f0ef7f55b38 <thread_status+184>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7f0ef7f55b38 <thread_status+184>, mutex=0x7f0ef7f55b10 <thread_status+144>) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f0ef64bd43b in blas_thread_server () from target:/var/www/test-proj/venv2/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-15028c96.3.21.so
#6  0x00007f0f09e30b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007f0f09ec2a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7f0efc9ff640 (LWP 223085) "apache2"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f0ef7f55ae0 <thread_status+96>) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f0ef7f55ae0 <thread_status+96>) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f0ef7f55ae0 <thread_status+96>, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007f0f09e2fac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f0ef7f55a90 <thread_status+16>, cond=0x7f0ef7f55ab8 <thread_status+56>) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x7f0ef7f55ab8 <thread_status+56>, mutex=0x7f0ef7f55a90 <thread_status+16>) at ./nptl/pthread_cond_wait.c:627
#5  0x00007f0ef64bd43b in blas_thread_server () from target:/var/www/test-proj/venv2/lib/python3.10/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-15028c96.3.21.so
#6  0x00007f0f09e30b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#7  0x00007f0f09ec2a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7f0efe194640 (LWP 222995) "apache2"):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=139702369338944) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=139702369338944) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=139702369338944, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007f0f09dde476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007f0f09dc47f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007f0f04c9dbbe in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007f0f04ca924c in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00007f0f04ca8229 in ?? () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007f0f04ca8999 in __gxx_personality_v0 () from target:/lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007f0f04bf1c64 in ?? () from target:/lib/x86_64-linux-gnu/libgcc_s.so.1
#10 0x00007f0f04bf26bd in _Unwind_Resume () from target:/lib/x86_64-linux-gnu/libgcc_s.so.1
#11 0x00007f0f064d0315 in ?? () from target:/lib/x86_64-linux-gnu/libproj.so.22
#12 0x00007f0f0651a514 in osgeo::proj::common::Angle::Angle(double) () from target:/lib/x86_64-linux-gnu/libproj.so.22
#13 0x00007f0ee7b2e4a2 in ?? () from target:/var/www/test-proj/venv2/lib/python3.10/site-packages/pyproj/../pyproj.libs/libproj-8d95adc6.so.25.9.2.0
#14 0x00007f0ee7b2ffc9 in ?? () from target:/var/www/test-proj/venv2/lib/python3.10/site-packages/pyproj/../pyproj.libs/libproj-8d95adc6.so.25.9.2.0
#15 0x00007f0f0a0b547e in call_init (l=<optimized out>, argc=argc@entry=3, argv=argv@entry=0x7fff447f7c68, env=env@entry=0x558aad9e5b70) at ./elf/dl-init.c:70
#16 0x00007f0f0a0b5568 in call_init (env=0x558aad9e5b70, argv=0x7fff447f7c68, argc=3, l=<optimized out>) at ./elf/dl-init.c:33
#17 _dl_init (main_map=0x7f0ef0f5ffc0, argc=3, argv=0x7fff447f7c68, env=0x558aad9e5b70) at ./elf/dl-init.c:117
#18 0x00007f0f09f10c85 in __GI__dl_catch_exception (exception=exception@entry=0x0, operate=operate@entry=0x7f0f0a0bcf40 <call_dl_init>, args=args@entry=0x7f0efe18b9d0) at ./elf/dl-error-skeleton.c:182
#19 0x00007f0f0a0bcff6 in dl_open_worker (a=0x7f0efe18bb70) at ./elf/dl-open.c:808
#20 dl_open_worker (a=a@entry=0x7f0efe18bb70) at ./elf/dl-open.c:771
#21 0x00007f0f09f10c28 in __GI__dl_catch_exception (exception=exception@entry=0x7f0efe18bb50, operate=operate@entry=0x7f0f0a0bcf60 <dl_open_worker>, args=args@entry=0x7f0efe18bb70) at ./elf/dl-error-skeleton.c:208
#22 0x00007f0f0a0bd34e in _dl_open (file=<optimized out>, mode=-2147483646, caller_dlopen=0x7f0eff7dc650, nsid=-2, argc=3, argv=<optimized out>, env=0x558aad9e5b70) at ./elf/dl-open.c:883
#23 0x00007f0f09e2c6bc in dlopen_doit (a=a@entry=0x7f0efe18bde0) at ./dlfcn/dlopen.c:56
#24 0x00007f0f09f10c28 in __GI__dl_catch_exception (exception=exception@entry=0x7f0efe18bd40, operate=<optimized out>, args=<optimized out>) at ./elf/dl-error-skeleton.c:208
#25 0x00007f0f09f10cf3 in __GI__dl_catch_error (objname=0x7f0efe18bd98, errstring=0x7f0efe18bda0, mallocedp=0x7f0efe18bd97, operate=<optimized out>, args=<optimized out>) at ./elf/dl-error-skeleton.c:227
#26 0x00007f0f09e2c1ae in _dlerror_run (operate=operate@entry=0x7f0f09e2c660 <dlopen_doit>, args=args@entry=0x7f0efe18bde0) at ./dlfcn/dlerror.c:138
#27 0x00007f0f09e2c748 in dlopen_implementation (dl_caller=<optimized out>, mode=<optimized out>, file=<optimized out>) at ./dlfcn/dlopen.c:71
#28 ___dlopen (file=<optimized out>, mode=<optimized out>) at ./dlfcn/dlopen.c:81
#29 0x00007f0eff7dc650 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#30 0x00007f0eff79d3dd in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#31 0x00007f0eff6deb13 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#32 0x00007f0eff626766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#33 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#34 0x00007f0eff62b9b8 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#35 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#36 0x00007f0eff629eee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#37 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#38 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#39 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#40 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#41 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#42 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#43 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#44 0x00007f0eff696464 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#45 0x00007f0eff6967cf in _PyObject_CallMethodIdObjArgs () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#46 0x00007f0eff79f664 in PyImport_ImportModuleLevelObject () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#47 0x00007f0eff62a705 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#48 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#49 0x00007f0eff77194e in PyEval_EvalCode () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#50 0x00007f0eff771edd in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#51 0x00007f0eff6deb13 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#52 0x00007f0eff626766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#53 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#54 0x00007f0eff62b9b8 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#55 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#56 0x00007f0eff629eee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#57 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#58 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#59 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#60 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#61 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#62 0x00007f0eff696464 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#63 0x00007f0eff6967cf in _PyObject_CallMethodIdObjArgs () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#64 0x00007f0eff79f664 in PyImport_ImportModuleLevelObject () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#65 0x00007f0eff62a705 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#66 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#67 0x00007f0eff77194e in PyEval_EvalCode () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#68 0x00007f0eff771edd in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#69 0x00007f0eff6deb13 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#70 0x00007f0eff626766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#71 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#72 0x00007f0eff62b9b8 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#73 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#74 0x00007f0eff629eee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#75 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#76 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#77 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#78 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#79 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#80 0x00007f0eff696464 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#81 0x00007f0eff6967cf in _PyObject_CallMethodIdObjArgs () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#82 0x00007f0eff79f664 in PyImport_ImportModuleLevelObject () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#83 0x00007f0eff62a705 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#84 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#85 0x00007f0eff77194e in PyEval_EvalCode () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#86 0x00007f0eff771edd in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#87 0x00007f0eff6deb13 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#88 0x00007f0eff626766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#89 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#90 0x00007f0eff62b9b8 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#91 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#92 0x00007f0eff629eee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#93 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#94 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#95 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#96 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#97 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#98 0x00007f0eff696464 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#99 0x00007f0eff6967cf in _PyObject_CallMethodIdObjArgs () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#100 0x00007f0eff79f664 in PyImport_ImportModuleLevelObject () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#101 0x00007f0eff62a705 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#102 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#103 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#104 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#105 0x00007f0eff77194e in PyEval_EvalCode () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#106 0x00007f0eff771edd in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#107 0x00007f0eff6deb13 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#108 0x00007f0eff626766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#109 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#110 0x00007f0eff62b9b8 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#111 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#112 0x00007f0eff629eee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#113 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#114 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#115 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#116 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#117 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#118 0x00007f0eff696464 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#119 0x00007f0eff6967cf in _PyObject_CallMethodIdObjArgs () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#120 0x00007f0eff79f664 in PyImport_ImportModuleLevelObject () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#121 0x00007f0eff62a705 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#122 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#123 0x00007f0eff77194e in PyEval_EvalCode () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#124 0x00007f0eff771edd in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#125 0x00007f0eff6deb13 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#126 0x00007f0eff626766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#127 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#128 0x00007f0eff62b9b8 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#129 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#130 0x00007f0eff629eee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#131 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#132 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#133 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#134 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#135 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#136 0x00007f0eff696464 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#137 0x00007f0eff6967cf in _PyObject_CallMethodIdObjArgs () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#138 0x00007f0eff79f664 in PyImport_ImportModuleLevelObject () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#139 0x00007f0eff62a705 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#140 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#141 0x00007f0eff77194e in PyEval_EvalCode () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#142 0x00007f0eff771edd in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#143 0x00007f0eff6deb13 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#144 0x00007f0eff626766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#145 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#146 0x00007f0eff62b9b8 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#147 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#148 0x00007f0eff629eee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#149 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#150 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#151 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#152 0x00007f0eff62862e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#153 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#154 0x00007f0eff696464 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#155 0x00007f0eff6967cf in _PyObject_CallMethodIdObjArgs () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#156 0x00007f0eff79f664 in PyImport_ImportModuleLevelObject () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#157 0x00007f0eff62a705 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#158 0x00007f0eff77681f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#159 0x00007f0eff77194e in PyEval_EvalCode () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#160 0x00007f0eff79e63d in PyImport_ExecCodeModuleObject () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#161 0x00007f0eff79e7a2 in PyImport_ExecCodeModuleWithPathnames () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#162 0x00007f0effc07002 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#163 0x00007f0effc0817f in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#164 0x00007f0effc0f9c8 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#165 0x00007f0f09e30b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#166 0x00007f0f09ec2a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7f0efe995640 (LWP 222994) "apache2"):
#0  0x00007f0f09eb77ed in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7f0efe994dd0) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x00007f0f09feef39 in apr_sleep () from target:/lib/x86_64-linux-gnu/libapr-1.so.0
#2  0x00007f0effc0ff1a in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#3  0x00007f0f09e30b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#4  0x00007f0f09ec2a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f0eff196640 (LWP 222993) "apache2"):
#0  0x00007f0f09eb77ed in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7f0eff195d60) at ../sysdeps/unix/sysv/linux/select.c:69
#1  0x00007f0f09feef39 in apr_sleep () from target:/lib/x86_64-linux-gnu/libapr-1.so.0
#2  0x00007f0effc1011c in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#3  0x00007f0f09e30b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#4  0x00007f0f09ec2a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f0f09d23780 (LWP 222990) "apache2"):
#0  0x00007f0f09eb4d7f in __GI___poll (fds=0x7fff447f75b0, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f0f09fe742b in apr_poll () from target:/lib/x86_64-linux-gnu/libapr-1.so.0
#2  0x00007f0effc152d8 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#3  0x00007f0effc16f1f in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#4  0x00007f0effc1b6cf in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#5  0x0000558aacd3cb34 in ap_run_post_config ()
#6  0x0000558aacd1c5d4 in main ()

I hope it helps. And thanks for your time.

SteffenKr avatar May 30 '23 14:05 SteffenKr

This could be a hard one to work out. To try and isolate it from everything else, can you test with a WSGI hello world program of:

import pyproj

def application(environ, start_response):
    status = '200 OK'
    output = b'Hello World!'

    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]
    start_response(status, response_headers)

    return [output]

If that still fails, next step would be to edit [pyproj](https://github.com/pyproj4/pyproj/tree/main/pyproj) /__init__.py file directly in the virtual environment where it is installed and comment out all sub imports except the first one. Progressively add the imports until we find which one causes it to crash. This might help to give ideas.

As to the strack trace, it seems to indicate that the C++ code in proj library had an unhandled C++ exception. I don't understand how that could occur in osgeo::proj::common::Angle::Angle(double) as the code for it is quite simple (although nothing is simple in C++ because of inline template expansion nonsense).

In general problems with unexpected crashes in wrapped libraries under an embedded system can come about for a few reasons.

  • Symbol clashes where a wrapped library doesn't prefix public functions/variables with a unique name and the name is already in use by something previously dynamically loaded into Apache. This can result in wrong function being called at runtime. For C++ the chances of this should be less due to C++ symbol name mangling.
  • Code not having permissions to access stuff and it doesn't cope with it well. This can be a problem with Apache since runs as Apache user and for Red Hat derived operating systems SELinux may block stuff.
  • Code having expectations of where in file system it runs. Under Apache default current working directory is usually root directory. You can try setting home option on WSGIDaemonProcess to location of source code or project instead.

Anyway, lets see if that most simple of hello world programs with just the problematic import causes anything to fail.

GrahamDumpleton avatar May 30 '23 22:05 GrahamDumpleton