Recording Pypy Segfaults in Testing

[CoolCat467]

Gah, well see the segfaults don't exactly provide much information... But I agree that it's annoying. At this point ig we might as well make an issue here (on trio) to track things cause upstream pytest hasn't been able to solve this yet.

Originally posted by @A5rocks in https://github.com/python-trio/trio/issues/2930#issuecomment-1904173395

This issue is for recording segfaults during pypy runs in the github actions testing runner.

So far (also feel free to edit and add more): Windows x64 pypy 3.9 nightly: https://github.com/python-trio/trio/actions/runs/5705575738/job/15460400267?pr=2724 Ubuntu pypy-3.10-nightly: https://github.com/python-trio/trio/actions/runs/7326018868/job/19951219901#step:5:1197 Ubuntu pypy-3.9: https://github.com/python-trio/trio/actions/runs/7607032357/job/20713702030#step:5:1186 Ubuntu pypy-3.9-nightly: https://github.com/python-trio/trio/actions/runs/7646760307/job/20836270507#step:5:1188

Not segfault but another error during Ubuntu pypy-3.9: TypeError: expected some sort of stmt, got '@py_assert1' https://github.com/python-trio/trio/actions/runs/7054572593/job/19203654087#step:5:1147

[jakkdl]

another instance of the ubuntu pypy-3.9: https://github.com/python-trio/trio/actions/runs/7698138955/job/20976753073#step:5:1193

[A5rocks]

Are we sure the Windows segfault is this?

[A5rocks]

and another ubuntu pypy-3.9 one: https://github.com/python-trio/trio/actions/runs/7703752769/job/20994747752#step:5:1186

[CoolCat467]

Are we sure the Windows segfault is this?

No, the logs expired for that one. EDIT: Issue comment where I mentioned this particular segfault: https://github.com/python-trio/trio/pull/2724#issuecomment-1657099912

[jakkdl]

https://github.com/python-trio/trio/actions/runs/7712303460/job/21019559589?pr=2933

I think this is just a random network error?

macOS pypy-3.9-nightly

Run codecov/codecov-action@v3
  with:
    directory: empty
    token: 87cefb17-c44b-4f2f-8b30-1fff5769ce46
    name: macOS (pypy-3.9-nightly)
    flags: macOS,pypy-3.9-nightly
==> macos OS detected
Error: getaddrinfo ENOTFOUND uploader.codecov.io
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:118:26)

[jakkdl]

ubuntu pypy-3.9 https://github.com/python-trio/trio/actions/runs/7725935804/job/21061210801?pr=2941#step:5:1489

  Fatal Python error: Segmentation fault
  
  Stack (most recent call first, approximate line numbers):
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/jedi/api/completion.py", line 68 in filter_names
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/jedi/api/completion.py", line 141 in complete
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/jedi/api/__init__.py", line 194 in complete
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/jedi/api/helpers.py", line 469 in wrapper
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/trio/_tests/test_exports.py", line 245 in test_static_tool_sees_class_members
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 187 in pytest_pyfunc_call
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 1790 in runtest
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 160 in pytest_runtest_call
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 262 in <lambda>
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 318 in from_call
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 247 in call_runtest_hook
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 219 in call_and_report
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 119 in runtestprotocol
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 338 in pytest_runtestloop
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 321 in _main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 258 in wrap_session
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 135 in main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 185 in console_main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pytest/__main__.py", line 1 in <module>
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/execfile.py", line 169 in run
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 810 in do_run
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 608 in command_line
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 961 in main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/bin/coverage", line 3 in <module>
    File "<builtin>/app_main.py", line 1017 in execfile
    File "<builtin>/app_main.py", line 131 in run_toplevel
    File "<builtin>/app_main.py", line 737 in run_command_line
    File "<builtin>/app_main.py", line 1131 in entry_point
  ../../../../../../opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/trio/_tests/test_exports.py::test_static_tool_sees_class_members[jedi-trio.lowlevel] ::endgroup::
  ./ci.sh: line 151:  1981 Segmentation fault      (core dumped) COVERAGE_PROCESS_START=$(pwd)/../pyproject.toml coverage run --rcfile=../pyproject.toml -m pytest -ra --junitxml=../test-results.xml --run-slow ${INSTALLDIR} --verbose --durations=10 $flags

[A5rocks]

Oh I see I forgot to skip jedi on pypy for test_static_tool_sees_class_members (I fixed that for test_static_tool_sees_all_symbols)

[jakkdl]

another instance of the same ubuntu pypy-3.9 error: https://github.com/python-trio/trio/actions/runs/7886026350/job/21518402835#step:5:1142

succeeded on rerun. Looking closer at the logs, the error is from a runner using Runner image version 20240204.1.0, and the success one uses 20240212.1.0. Although 7 hours ago a 20240204.1.0 ubuntu pypy-3.9 run did succeed: https://github.com/python-trio/trio/actions/runs/7883080425/job/21509366109 so that's probably not a thing either.

[Zac-HD]

Recently fixed upstream, so I think we're done! https://github.com/pytest-dev/pytest/issues/11168#issuecomment-2023988486

https://www.pypy.org/posts/2024/03/fixing-bug-incremental-gc.html is well worth reading, too.

[jakkdl]

Great writeup! Very impressive debugging skillz

[CoolCat467]

We had another one in https://github.com/python-trio/trio/pull/2972, https://github.com/python-trio/trio/actions/runs/8579945946/job/23515500181#step:5:1232, have we updated pypy or anything?

[Zac-HD]

Looks like the latest version is PyPy 7.3.15, from Jan 15th - we're running this, and the fix just hasn't been released yet. From the post:

At the point of publishing this post, the fixes got merged to the 2.7/3.9/3.10 branches of PyPy, and will be part of the next release (v7.3.16).

Happy to leave this open until PyPy is released, but since we get patch updates automatically via GitHub actions I think we're basically done here.

[CoolCat467]

Looking at https://github.com/pypy/pypy/tags, it looks like v7.3.16 was released on Apr 20, anything we need to do on our end to update pypy version or are we good to close this now?

[Zac-HD]

I think we're good to close! Thanks for tracking 😁