blake3-py icon indicating copy to clipboard operation
blake3-py copied to clipboard

Add pypy to CI

Open oconnor663 opened this issue 2 years ago • 6 comments

~After https://github.com/PyO3/maturin/pull/596 ships.~ That fix has shipped.

oconnor663 avatar Jul 20 '21 06:07 oconnor663

My pypy_ci branch is getting started on this. It does run tests (though it's broken on Windows). Importantly, there seem to be some weird memory errors coming up:

https://github.com/oconnor663/blake3-py/runs/3115660174?check_suite_focus=true

Could be a PyO3 bug here? Some sort of PyPy quirkiness? I'll need to dig into this.

oconnor663 avatar Jul 20 '21 16:07 oconnor663

We are seeing errors in Github actions auto test with blake3 py 0.2.0

SmithSamuelM avatar Jul 21 '21 15:07 SmithSamuelM

I'm able to reproduce this error locally in a pypy3 virtualenv. I'll start here:

❯ pytest -vk strided
===================================== test session starts =====================================
platform linux -- Python 3.7.10[pypy-7.3.5-final], pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /tmp/venv.e5V/bin/pypy3
cachedir: .pytest_cache
rootdir: /home/jacko/blake3-py
collected 16 items / 15 deselected / 1 selected                                               

tests/test_blake3.py::test_strided_array_fails FAILED                                   [100%]

========================================== FAILURES ===========================================
__________________________________ test_strided_array_fails ___________________________________

    def test_strided_array_fails():
        unstrided = numpy.array([1, 2, 3, 4], numpy.uint8)
        strided = numpy.lib.stride_tricks.as_strided(unstrided,
                                                     shape=[2],
                                                     strides=[2])
>       assert bytes(strided) == bytes([1, 3])
E       AssertionError: assert b'\x01\x02' == b'\x01\x03'
E         At index 1 diff: b'\x02' != b'\x03'
E         Full diff:
E         - b'\x01\x03'
E         ?          ^
E         + b'\x01\x02'
E         ?          ^

tests/test_blake3.py:141: AssertionError
=================================== short test summary info ===================================
FAILED tests/test_blake3.py::test_strided_array_fails - AssertionError: assert b'\x01\x02' =...
============================== 1 failed, 15 deselected in 0.46s ===============================

oconnor663 avatar Jul 21 '21 16:07 oconnor663

I've filed https://github.com/numpy/numpy/issues/19536 related to the issue above. It doesn't appear to be blake3-specific. Not yet clear whether this explains other errors.

oconnor663 avatar Jul 21 '21 17:07 oconnor663

Now I doubt the Numpy issue is related. If I add this extremely simple test, it sometimes passes (all 100k iterations) and sometimes fails (after a variable number of iterations, usually ~200):

def test_simple():
    expected = "04e0bb39f30b1a3feb89f536c93be15055482df748674b00d26e5a75777702e9"
    for i in range(100_000):
        print(i)
        assert blake3(b"foo").hexdigest() == expected

The failures look like this:

malloc_consolidate(): invalid chunk size                                                                                                                                                       
Fatal Python error: Aborted                                                                                                                                                                    
                                                                                                                                                                                               
Stack (most recent call first, approximate line numbers):                                                                                                                                      
  File "/home/jacko/blake3-py/tests/test_blake3.py", line 27 in test_simple                                                                                                                    
  File "/tmp/venv.7pC/site-packages/_pytest/python.py", line 176 in pytest_pyfunc_call                                                                                                         
  File "/tmp/venv.7pC/site-packages/pluggy/callers.py", line 157 in _multicall                                                                                                                 
  File "/tmp/venv.7pC/site-packages/pluggy/manager.py", line 84 in <lambda>                                                                                                                    
  File "/tmp/venv.7pC/site-packages/pluggy/manager.py", line 90 in _hookexec                                                                                                                   
  File "/tmp/venv.7pC/site-packages/pluggy/hooks.py", line 272 in __call__                                                                                                                     
...

Something very fishy is going on with memory here.

oconnor663 avatar Jul 21 '21 17:07 oconnor663

I've narrowed this down to what I believe is a different bug in PyO3: https://github.com/PyO3/pyo3/issues/1736

oconnor663 avatar Jul 21 '21 20:07 oconnor663