Selftests sometimes fail on pypy: `TypeError: expected some sort of stmt, but got <_ast.Load object ...>`

[bluetech]

trafficstars

Example: https://github.com/pytest-dev/pytest/actions/runs/7408595776/job/20157160097

This doesn't always happen, maybe 50% of the time. Started in the last few days. I checked if the PyPy version changed during these days, but seems not (it's 3.8.16).

The error is:

 __________________ ERROR collecting testing/_py/test_local.py __________________
/opt/hostedtoolcache/PyPy/3.8.16/x64/lib/pypy3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1023: in _gcd_import
    ???
<frozen importlib._bootstrap>:1000: in _find_and_load
    ???
<frozen importlib._bootstrap>:984: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:680: in _load_unlocked
    ???
.tox/pypy3-xdist/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py:166: in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
.tox/pypy3-xdist/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py:350: in _rewrite_test
    co = compile(tree, strfn, "exec", dont_inherit=True)
E   TypeError: expected some sort of stmt, but got <_ast.Store object at 0x0000000002142880>

Sometimes it says TypeError: expected some sort of stmt, but got <_ast.Load object at 0x000000000365d040> instead.

[skirpichev]

I have same failure on older versions as well. E.g. 7.4.3: https://github.com/mpmath/mpmath/actions/runs/8030300102?pr=759

This happens for all tested in the mpmath pypy versions (3.8-3.10). But most often on 3.10.

[skirpichev]

Another example: https://github.com/pytest-dev/pytest/issues/11168#issuecomment-1714358183

@cfbolz, I don't see a bugreport in pypy issues. Is there any chance this might be related to pytest-dev/pytest#11168 (an fixed in pypy)?

[cfbolz]

@skirpichev it's not impossible that it's the same problem, yes. You can try a nightly PyPy build (or wait for the next release but that is still a bit out).

[skirpichev]

@cfbolz, thanks for a suggestion. I did tests for nightly builds (on 3.8-3.10) and, after ~20 re-runs, I got a TypeError on pypy3.8, which is similar to this issue:

============================= test session starts ==============================
platform linux -- Python 3.8.16[pypy-7.3.12-alpha], pytest-8.1.1, pluggy-1.4.0
mpmath backend: python
mpmath mp class: <mpmath.ctx_mp.MPContext object at 0x00000000018deb10>
mpmath version: 1.4.0a1.dev43+g534bc79
Python version: 3.8.16 (d8e6b37f5231c7b59b2b639a651dd696a3732cc3, Jan 09 2023, 00:30:15)
[PyPy 7.3.12-alpha0 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]
rootdir: /home/runner/work/mpmath/mpmath
configfile: pyproject.toml
testpaths: mpmath, docs
plugins: cov-5.0.0, xdist-3.5.0
created: 4/4 workers
4 workers [2294 items]

........................................................................ [  3%]
........................................................................ [  6%]
........................................................................ [  9%]
........................................................................ [ 12%]
........................................................................ [ 15%]
........................................................................ [ 18%]
........................................................................ [ 21%]
........................................................................ [ 25%]
........................................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 34%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 47%]
........................................................................ [ 50%]
........................................................................ [ 53%]
........................................................................ [ 56%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 65%]
........................................................................ [ 69%]
........................................................................ [ 72%]
......................................................................... [ 75%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 87%]
........................................x............................... [ 91%]
.......................................x................................ [ 94%]
...................................s.................................... [ 97%]
.............................................................            [100%]
==================================== ERRORS ====================================
_______________ ERROR collecting mpmath/tests/test_basic_ops.py ________________
../../_temp/d367cbdc-8911-4ebe-aab6-2151925ee5c2/pypy-c-jit-106755-d8e6b37f5231-linux64/lib/pypy3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1023: in _gcd_import
    ???
<frozen importlib._bootstrap>:1000: in _find_and_load
    ???
<frozen importlib._bootstrap>:984: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:680: in _load_unlocked
    ???
../../_temp/d367cbdc-8911-4ebe-aab6-2151925ee5c2/pypy-c-jit-106755-d8e6b37f5231-linux64/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py:169: in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
../../_temp/d367cbdc-8911-4ebe-aab6-2151925ee5c2/pypy-c-jit-106755-d8e6b37f5231-linux64/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py:353: in _rewrite_test
    co = compile(tree, strfn, "exec", dont_inherit=True)
E   TypeError: expected some sort of stmt, but got <_ast.Attribute object at 0x00007f101d7a6920>
============================= slowest 20 durations =============================
34.04s call     mpmath/tests/test_functions2.py::test_bessel_zeros_extra
28.52s call     mpmath/tests/test_torture.py::test_asymp[<lambda>-150-False23]
16.64s call     mpmath/functions/hypergeometric.py::mpmath.functions.hypergeometric.hyper
11.94s call     mpmath/tests/test_extra_zeta.py::test_zetazero[241389216-97490234.22767118]
10.72s call     mpmath/functions/zeta.py::mpmath.functions.zeta.secondzeta
9.91s call     mpmath/tests/test_elliptic.py::test_elliptic_integrals
9.25s call     mpmath/ctx_fp.py::mpmath.ctx_fp.FPContext.bessely
8.75s call     mpmath/functions/hypergeometric.py::mpmath.functions.hypergeometric.hyp3f2
8.59s call     mpmath/ctx_fp.py::mpmath.ctx_fp.FPContext.siegelz
8.44s call     mpmath/calculus/extrapolation.py::mpmath.calculus.extrapolation.nsum
8.39s call     mpmath/tests/test_torture.py::test_asymp[<lambda>-150-False9]
7.87s call     mpmath/functions/zetazeros.py::mpmath.functions.zetazeros.nzeros
7.58s call     mpmath/tests/test_torture.py::test_asymp[f_wrapped-1500-False1]
7.53s call     mpmath/tests/test_quad.py::test_quadosc
7.50s call     mpmath/calculus/quadrature.py::mpmath.calculus.quadrature.QuadratureMethods.quad
7.45s call     mpmath/tests/test_torture.py::test_asymp[f_wrapped-90-False]
7.27s call     mpmath/tests/test_torture.py::test_asymp[f-5000-True1]
7.22s call     mpmath/tests/test_torture.py::test_asymp[<lambda>-150-False27]
7.00s call     mpmath/tests/test_torture.py::test_asymp[<lambda>-150-False37]
6.74s call     mpmath/tests/test_torture.py::test_asymp[<lambda>-150-False36]
=========================== short test summary info ============================
ERROR mpmath/tests/test_basic_ops.py - TypeError: expected some sort of stmt, but got <_ast.Attribute object at 0x00007f101d7a6920>
======= 2291 passed, 1 skipped, 2 xfailed, 1 error in 203.43s (0:03:23) ========

Summary page: https://github.com/mpmath/mpmath/actions/runs/8491432157?pr=772

~~Maybe this fix isn't merged yet to the 3.8 branch (as it was stated in your blog post)? If so,~~ I'll continue testing.

[cfbolz]

Yes, the 3.8 variants are not maintained any more, you need to try 3.9 or 3.10. Thanks for trying this out!

[skirpichev]

Well, I did ~50 re-runs of mpmath's tests on pypy3.9 and pypy3.10 - without any failures. Given the previous statistics, probably there is a chance that this issue was fixed in PyPy's nightly builds. I'll switch regular tests for pypy3.9/10 to them.

Thanks for bugfix!

[cfbolz]

Well, I did ~50 re-runs of mpmath's tests on pypy3.9 and pypy3.10 - without any failures.

thank you for trying this, that's very valuable feedback!

[webknjaz]

Sometimes it says TypeError: expected some sort of stmt, but got <_ast.Load object at 0x000000000365d040> instead.

During the sprints, we were seeing 3–4 variants of different AST object reprs there. Hopefully, bumping PyPy in CI will fix this. Ronny suggested this, but nobody got to actually doing it...

[webknjaz]

FTR @LilyFoote linked https://www.pypy.org/posts/2024/03/fixing-bug-incremental-gc.html in https://github.com/pytest-dev/pytest/issues/11168#issuecomment-2023988486 but that reference never got cross-posted into this issue.

[webknjaz]

Action item for whoever gets to prepare the PR:

• [x] Update this to 3.9 https://github.com/pytest-dev/pytest/blob/51ee388/.github/workflows/test.yml#L153
• [x] Mention the above article in the contrib-type change note
• [x] Add a Resolves #11771 somewhere in the commit message or the PR description

[x612skm]

Working on this issue

[webknjaz]

Sometimes it says TypeError: expected some sort of stmt, but got <_ast.Load object at 0x000000000365d040> instead.

I even saw a case of NameError, which seems to have the same root cause:

 E   NameError: name '@py_assert2' is not defined

(https://github.com/pytest-dev/pytest/actions/runs/9746885084/job/26898335196#step:6:97)