big endian support

[andreasstieger]

Describe the bug

There seems to be an architecture specific build failure on s390x (zSystems). Version range is unknown, but at least 0.10.3...0.10.5, possibly longer.

Seen in https://build.opensuse.org/package/show/devel:libraries:c_c++/aws-c-common

Regression Issue

• [ ] Select this option if this issue appears to be a regression.

Expected Behavior

test / build pass

Current Behavior

[  282s] 265/449 Test #267: sba_threaded_reallocs ................................................Subprocess aborted***Exception:   0.04 sec
[  282s] Fatal error condition occurred in /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/tests/alloc_test.c:200: 0 == memcmp(alloc, &thread_data->thread_idx, 1)
[  282s] Exiting Application
[  282s] ################################################################################
[  282s] Stack trace:
[  282s] ################################################################################
[  282s] /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/build/libaws-c-common.so.1(aws_backtrace_print+0x68) [0x3ffae12d3a0]
[  282s] /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/build/libaws-c-common.so.1(aws_fatal_assert+0x46) [0x3ffae11aafe]
[  282s] /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/build/tests/aws-c-common-tests(+0x18a64) [0x2aa06598a64]
[  282s] /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/build/libaws-c-common.so.1(+0x3386a) [0x3ffae13386a]
[  282s] /lib64/libc.so.6(+0xa868a) [0x3ffadea868a]
[  282s] /lib64/libc.so.6(+0x1292f0) [0x3ffadf292f0]
[  282s] 
[  282s] 266/449 Test #270: default_threaded_reallocs ............................................Subprocess aborted***Exception:   0.02 sec
[  282s] Fatal error condition occurred in /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/tests/alloc_test.c:200: 0 == memcmp(alloc, &thread_data->thread_idx, 1)
[  282s] Exiting Application
[  282s] ################################################################################
[  282s] Stack trace:
[  282s] ################################################################################
[  282s] /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/build/libaws-c-common.so.1(aws_backtrace_print+0x68) [0x3ffa58ad3a0]
[  282s] /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/build/libaws-c-common.so.1(aws_fatal_assert+0x46) [0x3ffa589aafe]
[  282s] /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/build/tests/aws-c-common-tests(+0x18a64) [0x2aa12598a64]
[  282s] /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/build/libaws-c-common.so.1(+0x3386a) [0x3ffa58b386a]
[  282s] /lib64/libc.so.6(+0xa868a) [0x3ffa56a868a]
[  282s] /lib64/libc.so.6(+0x1292f0) [0x3ffa57292f0]
[  282s] 
[  282s]         Start 273: aligned_threaded_allocs_and_frees
[  282s]         Start 274: test_memtrace_none
[  282s] 267/449 Test #272: aligned_threaded_reallocs ............................................Subprocess aborted***Exception:   0.03 sec
[  282s] Fatal error condition occurred in /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/tests/alloc_test.c:200: 0 == memcmp(alloc, &thread_data->thread_idx, 1)
[  282s] Exiting Application
[  282s] ################################################################################
[  282s] Stack trace:
[  282s] ################################################################################
[  282s] /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/build/libaws-c-common.so.1(aws_backtrace_print+0x68) [0x3ff8262d3a0]
[  282s] /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/build/libaws-c-common.so.1(aws_fatal_assert+0x46) [0x3ff8261aafe]
[  282s] /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/build/tests/aws-c-common-tests(+0x18a64) [0x2aa11418a64]
[  282s] /home/abuild/rpmbuild/BUILD/aws-c-common-0.10.5/build/libaws-c-common.so.1(+0x3386a) [0x3ff8263386a]
[  282s] /lib64/libc.so.6(+0xa868a) [0x3ff823a868a]
[  282s] /lib64/libc.so.6(+0x1292f0) [0x3ff824292f0]
[  282s] 
[...]
[  311s] 
[  311s] 99% tests passed, 3 tests failed out of 449
[  311s] 
[  311s] Total Test time (real) =  33.80 sec
[  311s] 
[  311s] The following tests FAILED:
[  311s] 	267 - sba_threaded_reallocs (Subprocess aborted)
[  311s] 	270 - default_threaded_reallocs (Subprocess aborted)
[  311s] 	272 - aligned_threaded_reallocs (Subprocess aborted)
[  311s] Errors while running CTest
[  311s] error: Bad exit status from /var/tmp/rpm-tmp.JkphOR (%check)
[  311s] 

Reproduction Steps

Build on s390x arch

https://build.opensuse.org/package/show/devel:libraries:c_c++/aws-c-common

Possible Solution

No response

Additional Information/Context

No response

aws-c-common version used

0.10.5

Compiler and version used

cc14-14.2.1

Operating System and version

openSUSE Tumbleweed, s390x

[DmitriyMusatkin]

Did it ever work? The failing tests are relatively new (maybe a year old), but the allocator behavior hasnt changed in a while. We dont officially support big-endian machines as we currently dont test on any, so i suspect allocator might be doing some big-endian unsafe assumptions,

[andreasstieger]

Build history shows this never worked, first added to the distro with 0.9.12

[jmklix]

This is something that we currently don't support in aws-c-common. We do have some tests in aws-checksums to make sure that works on big endian machines. Changing this to a feature request.

Please 👍 the original issue above if you would like this feature.

[sertonix]

As far as I could determine this is just a small issue in the test suite and doesn't affect the library. A fix could look like this:

--- a/tests/alloc_test.c
+++ b/tests/alloc_test.c
@@ -197,12 +197,12 @@
         size_t old_size = size;
         size = rand() % 1024;
         if (old_size) {
-            AWS_FATAL_ASSERT(0 == memcmp(alloc, &thread_data->thread_idx, 1));
+            AWS_FATAL_ASSERT(*(uint8_t*)alloc == (uint8_t)thread_data->thread_idx);
         }
         AWS_FATAL_ASSERT(0 == aws_mem_realloc(test_allocator, &alloc, old_size, size));
         /* If there was a value, make sure it's still there */
         if (old_size && size) {
-            AWS_FATAL_ASSERT(0 == memcmp(alloc, &thread_data->thread_idx, 1));
+            AWS_FATAL_ASSERT(*(uint8_t*)alloc == (uint8_t)thread_data->thread_idx);
         }
         if (size) {
             memset(alloc, (int)thread_data->thread_idx, size);

[tollsimy]

@sertonix I can confirm that PR #1218 resolves all test failures on my end. I'm particularly interested in this fix to help get the package building on s390x and hopefully included for that architecture in Fedora. In our case, #1218 together with https://github.com/awslabs/aws-c-cal/pull/225 makes awscli2 pkg build successful on s390x.

[sertonix]

@tollsimy To avoid duplicate work you should probably take a look at https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/89031

[tollsimy]

@sertonix It looks like our work isn’t overlapping at the moment, thanks for the heads-up! You might also want to take a look at: https://github.com/awslabs/aws-c-cal/pull/225, we had some issues with aws-c-cal too.