PySyncObj icon indicating copy to clipboard operation
PySyncObj copied to clipboard

test_logCompactionRegressionTest1 randomly sees a silent EOFError

Open JustAnotherArchivist opened this issue 7 years ago • 1 comments

test_logCompactionRegressionTest1 sometimes produces an EOFError in the pickle module, but this error does not have any influence on the test result, i.e. the test still passes even when the error occurs. The error is therefore only visible when running pytest with the -s option to disable output capturing. Here's the test output for pytest -v -s -k 'test_logCompactionRegressionTest1' test_syncobj.py when the test fails (paths redacted):

 test_syncobj.py::test_logCompactionRegressionTest1 [2018-07-31 12:48:20,997 syncobj.py:1325 ERROR]  failed to load full dump
 Traceback (most recent call last):
   File ".../pysyncobj/pickle.py", line 54, in load
     return pickle.load(file)
 EOFError: Ran out of input
 
 During handling of the above exception, another exception occurred:
 
 Traceback (most recent call last):
   File ".../pysyncobj/syncobj.py", line 1295, in __loadDumpFile
     data = self.__serializer.deserialize()
   File ".../pysyncobj/serializer.py", line 106, in deserialize
     return pickle.load(g)
   File ".../pysyncobj/pickle.py", line 58, in load
     return pickle._load(file)
   File ".../lib/python3.6/pickle.py", line 1551, in _load
     encoding=encoding, errors=errors).load()
   File ".../lib/python3.6/pickle.py", line 1048, in load
     raise EOFError
 EOFError
 PASSED

Looking at the test code, I noticed that the test compares the raftLog object before and after deserialisation. However, raftLog is not the actual log data but just a MemoryJournal object. Since that class doesn't implement an __eq__ method, the comparison is done based on the object identity. And as raftLog refers to the same object before and after deserialisation, this will always be true, even if the contents of the journal were modified inbetween. In this case, that doesn't happen: o1._SyncObj__raftLog[:] is identical before and after deserialisation ([(b'\x01', 1, 0), (b'\x01', 2, 2)]). So there must be something else going on. But the test should still be fixed to compare the data before and after rather than the object against itself (by assigning o1._SyncObj__raftLog[:] instead of o1._syncObj__raftLog to logAfterCompaction and logAfterDeserialize).

This may be related to #85 as it's seemingly also caused by the serialiser and only happens occasionally.

JustAnotherArchivist avatar Jul 31 '18 13:07 JustAnotherArchivist

By the way, the same issue of comparing raftLog objects instead of the log contents is also present in test_randomTest1, but it's only used for a debugging print there, so that's not as important. (It will always print False though because each SyncObj instance has its own journal object.)

JustAnotherArchivist avatar Aug 01 '18 10:08 JustAnotherArchivist