raft icon indicating copy to clipboard operation
raft copied to clipboard

Test failed for armhfp: test_uv_load.c:33: assertion failed: status == 0 (18 == 0)

Open ganto opened this issue 3 years ago • 7 comments

Since the release of raft-0.11.3 the Fedora builds (COPR and Koji) for the armhfp architecture fail the test suite with the following errors:

Running test suite with seed 0xf1d4103e...
truncate/closingMultiple                                    [ OK    ] [ 0.01876796 / 0.01873492 CPU ]
truncate/closing                                            [ OK    ] [ 0.02283790 / 0.01840082 CPU ]
truncate/multiplePending                                    [ ERROR ]
Error: test/integration/test_uv_truncate.c:32: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
truncate/pendingAppend                                      [ ERROR ]
Error: test/integration/test_uv_truncate.c:32: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
truncate/partialSegment                                     [ ERROR ]
Error: test/integration/test_uv_truncate.c:32: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
truncate/sameAsLastIndex                                    [ ERROR ]
Error: test/integration/test_uv_truncate.c:32: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
truncate/wholeSegment                                       [ ERROR ]
Error: test/integration/test_uv_truncate.c:32: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
snapshot_put/installMultipleAppendEntriesInBetween          [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
snapshot_put/installMultipleWithoutPreviousEntries          [ OK    ] [ 0.03585798 / 0.03152932 CPU ]
snapshot_put/installWithoutPreviousEntries                  [ OK    ] [ 0.03016909 / 0.02784379 CPU ]
snapshot_put/installNoCompression                           [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
snapshot_put/install                                        [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
snapshot_put/entriesMoreThanTrailing                        [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
snapshot_put/entriesLessThanTrailing                        [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
snapshot_put/first                                          [ OK    ] [ 0.03587367 / 0.03149664 CPU ]
tcp_listen/handshake                                        
  n=8                                                       [ OK    ] [ 0.00232383 / 0.00232287 CPU ]
  n=16                                                      [ OK    ] [ 0.00228932 / 0.00228460 CPU ]
  n=24                                                      [ OK    ] [ 0.00250800 / 0.00250460 CPU ]
  n=32                                                      [ OK    ] [ 0.00247014 / 0.00245287 CPU ]
[...]
load/openSegmentWithBadFormat                               [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentWithZeroFormatAndThenData                   [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentWithNoAccessPermission                      [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/closedSegmentWithBadFormat                             [ OK    ] [ 0.00713523 / 0.00708564 CPU ]
load/emptyClosedSegment                                     [ OK    ] [ 0.00624074 / 0.00622865 CPU ]
load/closedSegmentWithBadIndex                              [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/closedSegmentWithCorruptedBatchData                    [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/closedSegmentWithCorruptedBatchHeader                  [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentWithIncompleteBatchData                     [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentWithIncompleteBatchHeader                   [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentWithIncompletePreamble                      [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentWithIncompleteFormat                        [ OK    ] [ 0.00757212 / 0.00655343 CPU ]
load/closedSegmentWithEntriesPastSnapshot                   [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/nonContiguousClosedSegments                            [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/closedSegmentsOverlappingWithSnapshot                  [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentNoClosedSegmentsSnapshotPresent             [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentWithEntriesBehindSnapshot                   [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/closedSegmentWithInconsistentFilename                  [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentWithEntriesPastSnapshot                     [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/closedSegmentWithEntriesBehindSnapshot                 [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/orphanedSnapshotFiles                                  [ OK    ] [ 0.04432707 / 0.04063180 CPU ]
load/emptySnapshot                                          [ OK    ] [ 0.04160712 / 0.03908186 CPU ]
load/manySnapshots                                          [ OK    ] [ 0.04186135 / 0.04186266 CPU ]
load/onlyOneSnapshot                                        [ OK    ] [ 0.03653829 / 0.03653956 CPU ]
load/openSegment                                            [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/secondOpenSegmentIsAllZeros                            [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/twoOpenSegments                                        [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentWithIncompleteFirstBatch                    [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentWithIncompleteBatch                         [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentWithNonZeroData                             [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/bothOpenAndClosedSegments                              [ ERROR ]
Error: test/integration/test_uv_load.c:33: assertion failed: status == 0 (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
load/openSegmentWithTrailingZeros                           [ OK    ] [ 0.00842839 / 0.00717770 CPU ]
load/emptyOpenSegment                                       [ OK    ] [ 0.00590746 / 0.00583415 CPU ]
load/removeUnusableFiles                                    
  filename=tmp-0000000001221212-0000000001221217            [ OK    ] [ 0.00414280 / 0.00411250 CPU ]
  filename=tmp-snapshot-15-8260687-512469866                [ OK    ] [ 0.00420739 / 0.00417480 CPU ]
  filename=snapshot-525-43326736-880259052                  [ OK    ] [ 0.00577649 / 0.00570792 CPU ]
  filename=snapshot-999-13371337-880259052.meta             [ OK    ] [ 0.00563636 / 0.00562175 CPU ]
  filename=snapshot-20-8260687-512469866                    [ OK    ] [ 0.00570946 / 0.00569122 CPU ]
  filename=snapshot-88-8260687-512469866.meta               [ OK    ] [ 0.00562086 / 0.00560566 CPU ]
  filename=snapshot-88-8260999-512469866.meta               [ OK    ] [ 0.00562774 / 0.00561403 CPU ]
  filename=tmp-snapshot-88-8260999-512469866.meta           [ OK    ] [ 0.00413958 / 0.00412576 CPU ]
  filename=tmp-snapshot-33-8260687-512469866                [ OK    ] [ 0.00412509 / 0.00411161 CPU ]
  filename=snapshot-33-8260687-512469866.meta               [ OK    ] [ 0.00561378 / 0.00560014 CPU ]
  filename=tmp-metadata1                                    [ OK    ] [ 0.00405900 / 0.00405046 CPU ]
  filename=tmp-metadata2                                    [ OK    ] [ 0.00409677 / 0.00408856 CPU ]
  filename=tmp-open1                                        [ OK    ] [ 0.00391330 / 0.00390449 CPU ]
  filename=tmp-open13                                       [ OK    ] [ 0.00396583 / 0.00395651 CPU ]
load/ignoreUnknownFiles                                     
  filename=garbage                                          [ OK    ] [ 0.00478334 / 0.00477409 CPU ]
  filename=0000000000000000000000000001-00000000001garbage  [ OK    ] [ 0.00539432 / 0.00534071 CPU ]
  filename=open-1garbage                                    [ OK    ] [ 0.00534687 / 0.00533303 CPU ]
load/emptyDir                                               [ OK    ] [ 0.00344228 / 0.00341221 CPU ]
bootstrap/termIsNonZero                                     [ OK    ] [ 0.00252379 / 0.00252095 CPU ]
bootstrap/pristine                                          [ OK    ] [ 0.00245411 / 0.00244572 CPU ]
append/barrierNoOpenSegments                                [ ERROR ]
Error: test/integration/test_uv_append.c:620: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/barrierOpenSegments                                  [ ERROR ]
Error: test/integration/test_uv_append.c:620: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/ioSetupError                                         [ ERROR ]
Error: test/lib/aio.c:53: assertion failed: (*__errno_location ()) == 11 (38 == 11)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/currentSegment                                       [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/cancelPrepareRequest                                 [ SKIP  ]
append/removeSegmentUponClose                               [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/closeDuringWrite                                     [ SKIP  ]
append/oom                                                  
  heap-fault-delay=1, heap-fault-repeat=1                   [ OK    ] [ 0.00022372 / 0.00022416 CPU ]
append/writeError                                           [ SKIP  ]
append/noSpaceResolved                                      
  dir-fs=tmpfs                                              [ SKIP  ]
append/noSpaceUponWrite                                     
  dir-fs=tmpfs                                              [ SKIP  ]
append/noSpaceUponPrepareSpare                              
  dir-fs=tmpfs                                              [ SKIP  ]
append/noSpaceUponPrepareCurrent                            
  dir-fs=tmpfs                                              [ SKIP  ]
append/cancel                                               [ OK    ] [ 0.02173987 / 0.01925635 CPU ]
append/counter                                              [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/prepareClosing                                       [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/truncateClosing                                      [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/truncate                                             [ SKIP  ]
append/resizeArena                                          [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/wait                                                 [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/batch                                                [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/exceedBlock                                          [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/matchBlock                                           [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/fitBlock                                             [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/severalBig                                           [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/secondBig                                            [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/firstBig                                             [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/finalizeSegment                                      [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/prepareSegments                                      [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
append/first                                                [ ERROR ]
Error: test/integration/append_helpers.h:20: assertion failed: status == result->status (18 == 0)
qemu: uncaught target signal 6 (Aborted) - core dumped
Error: child killed by signal 6 (Aborted)
init/metadataOneAndTwoSameVersion                           [ OK    ] [ 0.00589842 / 0.00490550 CPU ]
init/metadataOneBadVersion                                  [ OK    ] [ 0.00427882 / 0.00425717 CPU ]
init/metadataOneBadFormat                                   [ OK    ] [ 0.00445705 / 0.00444844 CPU ]
init/metadataOneTooShort                                    [ OK    ] [ 0.00517933 / 0.00516549 CPU ]
init/noSpace                                                
  dir-fs=tmpfs                                              [ SKIP  ]
init/dirNotAccessible                                       [ OK    ] [ 0.00121192 / 0.00121205 CPU ]
init/dirDoesNotExist                                        [ OK    ] [ 0.00110383 / 0.00109958 CPU ]
init/oom                                                    
  heap-fault-delay=1, heap-fault-repeat=1                   [ SKIP  ]
init/dirTooLong                                             [ OK    ] [ 0.00019734 / 0.00019794 CPU ]
109 of 163 (67%) tests successful, 11 (6%) test skipped.
FAIL test/integration/uv (exit status: 1)

I believe this errors are all related as they all have a status = 18 instead of 0.

Note that the built is run on a x86_64 machine with arm emulation through qemu.

ganto avatar Feb 05 '22 16:02 ganto

Thanks, if by any chance you are willing to bisect to where the issue was introduced, I would be very grateful!

MathieuBordere avatar Feb 05 '22 17:02 MathieuBordere

Ok, I can reproduce it locally. It will take a while.

ganto avatar Feb 06 '22 01:02 ganto

Bisect returned commit a520dd6601ee7f9bb1662816f494b9ff9ad0a257 (part of #244) to be the first bad commit:

a520dd6601ee7f9bb1662816f494b9ff9ad0a257 is the first bad commit
commit a520dd6601ee7f9bb1662816f494b9ff9ad0a257
Author: Mathieu Borderé <[email protected]>
Date:   Thu Nov 4 14:27:51 2021 +0100

    test runner: Define order of constructors.
    
    On ppc64le they would run in the reverse order than on the other arches.
    
    Signed-off-by: Mathieu Borderé <[email protected]>

 test/lib/runner.h | 66 +++++++++++++++++++++++++++----------------------------
 1 file changed, 33 insertions(+), 33 deletions(-)

ganto avatar Feb 12 '22 13:02 ganto

If I simply revert this commit I can still successfully build and run the test suite on ppc64le (through qemu) as well as all other architectures x86_64, i686, armv7hl and aarch64, at least locally.

ganto avatar Feb 12 '22 15:02 ganto

Thanks a lot for investigating, I'll try to reproduce, maybe qemu does some magic with __attribute((constructor)) too and the priority values I have chosen interfere with qemu's chosen values.

MathieuBordere avatar Feb 12 '22 17:02 MathieuBordere

I with this commit reverted I was also able to build it on Koji so raft 0.11.3 can finally find it's way into Fedora.

ganto avatar Feb 12 '22 20:02 ganto

Ah yeah, if building/testing through cross compile with qemu-user-static you may get some pretty odd behaviors.

All our own build and test infrastructure run natively on arm64 which likely explains why we didn't see any of this.

stgraber avatar Feb 12 '22 21:02 stgraber

I'll close this as @ganto has found a solution that works for him.

MathieuBordere avatar Jun 29 '23 15:06 MathieuBordere