psutil
psutil copied to clipboard
giampaolo
[OpenBSD] segfault from `psutil.Process().cpu_affinity`
Summary
- OS: OpenBSD 6.9
- Architecture: PowerPC
- Psutil version: (5, 4, 3)
- Python version: Python 3.8.8
- Type: core
Description
Calling psutil.Process().cpu_affinity seg faults on OpenBSD 6.9:
builder3$ python3
Python 3.8.8 (default, Apr 19 2021, 14:29:36)
[Clang 10.0.1 ] on openbsd6
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.version_info
(5, 4, 3)
>>> psutil.Process().cpu_affinity()
Segmentation fault (core dumped)
From the comments here https://github.com/giampaolo/psutil/blob/master/psutil/_psutil_bsd.c CPU affinity is not supported on OpenBSD. OK, fine. But is it necessary to seg fault? Perhaps just doing nothing would be better?
This was found from: https://github.com/marxin/cvise/issues/68
Mmm that is weird. I read comments to marxin/cvise#68:
builder3$ python3
Python 3.8.8 (default, Apr 19 2021, 14:29:36)
[Clang 10.0.1 ] on openbsd6
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.version_info
(5, 4, 3)
>>> psutil.cpu_count(logical=False)
>>> psutil.cpu_count(logical=True)
2
>>> psutil.Process().cpu_affinity()
Segmentation fault (core dumped)
builder3$
I suspect the culprit here is not psutil.Process().cpu_affinity(), it's cpu_count. Basically the output you see is lying.
As an experiment, try to do import psutil; psutil.Process().cpu_affinity() and I expect you'll see an AttributeError exception (because cpu_affinity() on OpenBSD is not even exposed).
Also (if my assumption is correct), try calling cpu_count(logical=False/True) in a loop and eventually you should get a segfault.
Also I see:
>>> psutil.version_info
(5, 4, 3)
That's quite old. You should try with latest psutil version.
So I've updated the machine to OpenBSD 7.0 (current newest). That got me a slightly newer python, but the same crash:
builder3$ python3
Python 3.8.12 (default, Sep 28 2021, 03:58:03)
[Clang 11.1.0 ] on openbsd7
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.version_info
(5, 4, 3)
>>> psutil.Process().cpu_affinity()
Segmentation fault (core dumped)
Here's the backtrace:
builder3$ gdb python3 -c python3.8.core
(gdb) bt
#0 0xdd2ae530 in do_mkvalue () from /usr/local/lib/libpython3.8.so.0.0
#1 0xdd2ae78c in do_mktuple () from /usr/local/lib/libpython3.8.so.0.0
#2 0xdd2add84 in do_mkvalue () from /usr/local/lib/libpython3.8.so.0.0
#3 0xdd2acec8 in Py_BuildValue () from /usr/local/lib/libpython3.8.so.0.0
#4 0xc9979c38 in psutil_proc_oneshot_info () from /usr/local/lib/python3.8/site-packages/psutil/_psutil_bsd.cpython-38.so
#5 0xdd156dac in cfunction_call_varargs () from /usr/local/lib/libpython3.8.so.0.0
#6 0xdd157d5c in PyCFunction_Call () from /usr/local/lib/libpython3.8.so.0.0
#7 0xdd156404 in _PyObject_MakeTpCall () from /usr/local/lib/libpython3.8.so.0.0
#8 0xdd269a5c in call_function () from /usr/local/lib/libpython3.8.so.0.0
#9 0xdd266b38 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#10 0xdd2609c8 in PyEval_EvalFrameEx () from /usr/local/lib/libpython3.8.so.0.0
#11 0xdd157218 in function_code_fastcall () from /usr/local/lib/libpython3.8.so.0.0
#12 0xdd1573dc in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#13 0xdd269998 in call_function () from /usr/local/lib/libpython3.8.so.0.0
#14 0xdd266bbc in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#15 0xdd26a868 in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.8.so.0.0
#16 0xdd157458 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#17 0xdd269998 in call_function () from /usr/local/lib/libpython3.8.so.0.0
#18 0xdd266b1c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#19 0xdd2609c8 in PyEval_EvalFrameEx () from /usr/local/lib/libpython3.8.so.0.0
#20 0xdd157218 in function_code_fastcall () from /usr/local/lib/libpython3.8.so.0.0
#21 0xdd1573dc in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#22 0xdd15697c in PyVectorcall_Call () from /usr/local/lib/libpython3.8.so.0.0
#23 0xdd156b08 in PyObject_Call () from /usr/local/lib/libpython3.8.so.0.0
#24 0xdd266ddc in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#25 0xdd26a868 in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.8.so.0.0
#26 0xdd157458 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#27 0xdd269998 in call_function () from /usr/local/lib/libpython3.8.so.0.0
#28 0xdd266b1c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#29 0xdd2609c8 in PyEval_EvalFrameEx () from /usr/local/lib/libpython3.8.so.0.0
#30 0xdd157218 in function_code_fastcall () from /usr/local/lib/libpython3.8.so.0.0
#31 0xdd1573dc in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#32 0xdd269998 in call_function () from /usr/local/lib/libpython3.8.so.0.0
#33 0xdd266b1c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#34 0xdd26a868 in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.8.so.0.0
#35 0xdd157458 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#36 0xdd269998 in call_function () from /usr/local/lib/libpython3.8.so.0.0
#37 0xdd266b1c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#38 0xdd26a868 in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.8.so.0.0
#39 0xdd157458 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.8.so.0.0
#40 0xdd1561c8 in _PyObject_FastCallDict () from /usr/local/lib/libpython3.8.so.0.0
#41 0xdd1582ac in _PyObject_Call_Prepend () from /usr/local/lib/libpython3.8.so.0.0
#42 0xdd1dec50 in slot_tp_init () from /usr/local/lib/libpython3.8.so.0.0
#43 0xdd1d82e0 in type_call () from /usr/local/lib/libpython3.8.so.0.0
#44 0xdd156404 in _PyObject_MakeTpCall () from /usr/local/lib/libpython3.8.so.0.0
#45 0xdd269a5c in call_function () from /usr/local/lib/libpython3.8.so.0.0
#46 0xdd266b38 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.8.so.0.0
#47 0xdd26a868 in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.8.so.0.0
#48 0xdd26079c in PyEval_EvalCode () from /usr/local/lib/libpython3.8.so.0.0
#49 0xdd2c0654 in run_mod () from /usr/local/lib/libpython3.8.so.0.0
#50 0xdd2be6f0 in PyRun_InteractiveOneObjectEx () from /usr/local/lib/libpython3.8.so.0.0
#51 0xdd2bdc30 in PyRun_InteractiveLoopFlags () from /usr/local/lib/libpython3.8.so.0.0
#52 0xdd2bdaa8 in PyRun_AnyFileExFlags () from /usr/local/lib/libpython3.8.so.0.0
#53 0xdd2e7ecc in Py_RunMain () from /usr/local/lib/libpython3.8.so.0.0
#54 0xdd2e8328 in pymain_main () from /usr/local/lib/libpython3.8.so.0.0
#55 0xdd2e83ec in Py_BytesMain () from /usr/local/lib/libpython3.8.so.0.0
#56 0x367c0a04 in main () from /usr/local/bin/python3
I tried calling psutil.cpu_count(logical=True) in a loop 50000 times and it just printed 2 every time.
That's quite old. You should try with latest psutil version.
It seems to be the newest one:
builder3$ doas pkg_delete py-psutil
py-psutil-5.4.3p5: ok
builder3$ doas pkg_add py-psutil
quirks-4.53 signed on 2021-10-16T19:30:27Z
py-psutil-5.4.3p5: ok
Yet I see here: https://openports.se/sysutils/py-psutil a 5.8.0. I wonder if it just hasn't been built for PowerPC yet...?
Upgrade successful, crash remains:
builder3$ python3
Python 3.8.12 (default, Sep 28 2021, 03:58:03)
[Clang 11.1.0 ] on openbsd7
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.version_info
(5, 8, 0)
>>> psutil.Process().cpu_affinity()
Segmentation fault (core dumped)
I tried calling psutil.cpu_count(logical=True) in a loop 50000 times and it just printed 2 every time.
I think the problem here is psutil.Process() (aka you can avoid the ".cpu_affinity()" bit).
When Process() class is instantiated create_time() is invoked. On BSD, process create_time() is fetched by this big function:
https://github.com/giampaolo/psutil/blob/0e15b4890a1d84f2aa800b745fdb0642d2ee966d/psutil/_psutil_bsd.c#L184
Some time ago this same C function was causing a segfault: https://github.com/giampaolo/psutil/commit/842a50538fb518291079ab5f245cca4c63813cb2. That is fixed in 5.8.0 though, so this must be a new bug.
>>> psutil.Process().cpu_affinity() Segmentation fault (core dumped)
Can you please open the core file in gdb and investigate the line of code where it crashes? Or you may run valgrind that should dump a proper back-trace.
I think the problem here is psutil.Process() (aka you can avoid the ".cpu_affinity()" bit).
Ah ha! Yes, you are correct.
I've opened the core file in gdb, but it shows no source line numbers, presumably this is a release build, not a debug build. Here's a bit of info:
(gdb) frame 4
#4 0xb17382dc in psutil_proc_oneshot_info () from /home/builder/.local/lib/python3.8/site-packages/psutil/_psutil_bsd.cpython-38.so
0xb17382c4 <psutil_proc_oneshot_info+680> fsub f0,f8,f0
0xb17382c8 <psutil_proc_oneshot_info+684> fadd f4,f9,f0
0xb17382cc <psutil_proc_oneshot_info+688> fadd f2,f5,f2
0xb17382d0 <psutil_proc_oneshot_info+692> fadd f3,f7,f3
0xb17382d4 <psutil_proc_oneshot_info+696> fmr f5,f4
0xb17382d8 <psutil_proc_oneshot_info+700> bl 0xb173bd0c <00008000.got2.plt_pic32.Py_BuildValue>
•0xb17382dc <psutil_proc_oneshot_info+704> mr r27,r3
0xb17382e0 <psutil_proc_oneshot_info+708> lwz r3,0(r29)
0xb17382e4 <psutil_proc_oneshot_info+712> addi r3,r3,-1
0xb17382e8 <psutil_proc_oneshot_info+716> cmplwi r3,0
0xb17382ec <psutil_proc_oneshot_info+720> stw r3,0(r29)
0xb17382f0 <psutil_proc_oneshot_info+724> bne- 0xb17382fc <psutil_proc_oneshot_info+736>
valgrind does not support OpenBSD BTW.
I've never used gdb or Valgrind, so I'm not sure how to help with that.
If I were to debug this I would probably comment/disable most functionality of psutil_proc_oneshot_info function, and then gradually re-enable functionality line by line, until I bump into the segfault. If you GIT clone the repo, a quick way to test this "on the fly" after you edit psutil/_psutil_bsd.c would be:
python3 setup.py develop --user && python3 -c "import psutil; psutil.Process()"
Hmmm, this rabbit hole is getting pretty deep. I'm pretty far off what I'm trying to do, which is use cvise to auto-reduce a clang compiler crash. Its use of psutil.Process is not vital, and just commenting it out gets it "working" again.
I'll be setting up an Intel OpenBSD machine, and will at least be able to report if this issue occurs there too.
So on OpenBSD 7.0 on x86_64 (in VMWare), it works:
Python 3.8.12 (default, Sep 26 2021, 13:12:50)
[Clang 11.1.0 ] on openbsd7
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.version_info
(5, 8, 0)
>>> psutil.Process()
psutil.Process(pid=96111, name='python3.8', status='running', started='18:03:23')
>>> psutil.Process().cpu_affinity()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Process' object has no attribute 'cpu_affinity'
So I guess the issue is somehow PowerPC-specific... hopefully that's a clue.