ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

Numpy array test occasionaly raises segfault

Open vasil-pashov opened this issue 1 year ago • 1 comments

I think we have to sort out the GIL handling properly. This test,

    @pytest.mark.parametrize('n', range(1000))
    def test_append_empty_arrays_to_column(self, lmdb_version_store, array_type, n):
        df = pd.DataFrame({"col1": [np.array([1, 2, 3]).astype(array_type)]})
        lmdb_version_store.write("test_append_to_colum_with_empty_array", df)
        df_to_append = pd.DataFrame({"col1": [np.array([])]})
        lmdb_version_store.append("test_append_to_colum_with_empty_array", df_to_append)
        df_out = lmdb_version_store.read("test_append_to_colum_with_empty_array")
        df_target = pd.concat([df, df_to_append], ignore_index=True)
        assert_frame_equal(df_target, df_out.data)

that you added segfaults occasionally (hence the 1000 runs) in various places in Cython, indicating corruption of Python's datastructures. I've seen the test segfault when trying to run Python imports, or allocate tuples, for instance.

We've also seen it fail in the CI.

Originally posted by @poodlewars in https://github.com/man-group/ArcticDB/pull/819#discussion_r1434311648

Duplicate of: #1176

vasil-pashov avatar Jan 02 '24 08:01 vasil-pashov

Correct fix is probably #269

alexowens90 avatar Jan 08 '24 10:01 alexowens90