ilanschnell
Testsuite crashes on sparc64 due to unaligned access (Bus error)
The testsuite for bitarray crashes on sparc64 due to unaligned access (Bus error):
Successfully built bitarray-3.7.2-cp313-cp313-linux_sparc64.whl
I: pybuild plugin_pyproject:144: Unpacking wheel built for python3.13 with "installer" module
I: pybuild pybuild:334: mkdir -p /build/reproducible-path/python-bitarray-3.7.2/debian/buildhtml/doc && cp /build/reproducible-path/python-bitarray-3.7.2/README.rst /build/reproducible-path/python-bitarray-3.7.2/CHANGE_LOG /build/reproducible-path/python-bitarray-3.7.2/update_doc.py /build/reproducible-path/python-bitarray-3.7.2/debian/buildhtml; cp /build/reproducible-path/python-bitarray-3.7.2/doc/*.rst /build/reproducible-path/python-bitarray-3.7.2/debian/buildhtml/doc; cd /build/reproducible-path/python-bitarray-3.7.2/debian/buildhtml; PYTHONPATH=/build/reproducible-path/python-bitarray-3.7.2/.pybuild/cpython3_3.13_bitarray/build python3.13 update_doc.py; pandoc -r gfm -w html5 -o README.html -s --toc --metadata title="BitArray Documentation" README.rst; mv /build/reproducible-path/python-bitarray-3.7.2/debian/buildhtml/README.html /build/reproducible-path/python-bitarray-3.7.2
already up-to-date
dh_auto_test -a -O--buildsystem=pybuild
I: pybuild base:311: cd /build/reproducible-path/python-bitarray-3.7.2/.pybuild/cpython3_3.13_bitarray/build; python3.13 -c 'import bitarray; bitarray.test()'
..............Bus error
E: pybuild pybuild:389: test: plugin pyproject failed with: exit code=138: cd /build/reproducible-path/python-bitarray-3.7.2/.pybuild/cpython3_3.13_bitarray/build; {interpreter} -c 'import bitarray; bitarray.test()'
dh_auto_test: error: pybuild --test -i python{version} -p 3.13 returned exit code 13
Full build log available in: https://buildd.debian.org/status/fetch.php?pkg=python-bitarray&arch=sparc64&ver=3.7.2-1&stamp=1762556188&raw=0
This is usually easy to debug and fix with the help of GDB.
Access to a SPARC machine running both Solaris and Linux can be obtained through the GCC Compile Farm: https://gcc.gnu.org/wiki/CompileFarm
Thank you for the bug report! I have some ideas what might cause the bus error. Unfortunately I don't have time at the moment to look into this more, but I hope to look into it in December.
I really appreciate the open and kind reply. Thank you!
Let me try whether I can get you a backtrace real quick which might help down the problem faster.
Here is a quick backtrace from the core dump:
(gdb) bt
#0 0xfff8000101103fd4 in builtin_bswap64 (word=<optimized out>) at bitarray/bitarray.h:206
#1 shift_r8le (buff=buff@entry=0xba9a7c "\006\352\027ZY\356\357\221\f\356\241\351F\233\312\360\037\217\375\377\377[ym&\317\353\216\361\006\305d", n=<optimized out>,
k=k@entry=5) at bitarray/_bitarray.c:230
#2 0xfff8000101106920 in shift_r8 (self=<optimized out>, a=<optimized out>, b=149, k=5) at bitarray/_bitarray.c:293
#3 0xfff800010110a2f0 in copy_n (self=0xfff8000102871ac0, a=101, other=0xfff800010100aba0, b=0, n=1086) at bitarray/_bitarray.c:349
#4 0xfff800010110ec40 in setslice_bitarray (self=0xfff8000102871ac0, slice=<optimized out>, other=0xfff800010100aba0) at bitarray/_bitarray.c:2262
#5 assign_slice (self=0xfff8000102871ac0, slice=<optimized out>, value=<optimized out>) at bitarray/_bitarray.c:2346
#6 bitarray_ass_subscr (self=0xfff8000102871ac0, item=<optimized out>, value=<optimized out>) at bitarray/_bitarray.c:2567
#7 0x000000000015fc58 in PyObject_SetItem ()
#8 0x00000000002ea520 in _PyEval_EvalFrameDefault ()
#9 0x0000000000181aac in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)
The code in question is this one in shift_r8le():
#if HAVE_BUILTIN_BSWAP64 && PY_BIG_ENDIAN
*p = builtin_bswap64(*p);
*p <<= k;
*p = builtin_bswap64(*p);
#else
*p <<= k;
#endif
I managed to fix all cases of unaligned access. He is a very rough proof of concept patch:
Index: python-bitarray-3.7.2/bitarray/_bitarray.c
===================================================================
--- python-bitarray-3.7.2.orig/bitarray/_bitarray.c
+++ python-bitarray-3.7.2/bitarray/_bitarray.c
@@ -227,9 +227,12 @@ shift_r8le(unsigned char *buff, Py_ssize
while (w--) { /* shift in word-range(0, w) */
uint64_t *p = ((uint64_t *) buff) + w;
#if HAVE_BUILTIN_BSWAP64 && PY_BIG_ENDIAN
- *p = builtin_bswap64(*p);
- *p <<= k;
- *p = builtin_bswap64(*p);
+ uint64_t tmp;
+ memcpy(&tmp, p, sizeof(uint64_t));
+ tmp = builtin_bswap64(tmp);
+ tmp <<= k;
+ tmp = builtin_bswap64(tmp);
+ memcpy(p, &tmp, sizeof(uint64_t));
#else
*p <<= k;
#endif
@@ -261,7 +264,10 @@ shift_r8be(unsigned char *buff, Py_ssize
*p >>= k;
*p = builtin_bswap64(*p);
#else
- *p >>= k;
+ uint64_t tmp;
+ memcpy(&tmp, p, sizeof(uint64_t));
+ tmp >>= k;
+ memcpy(p, &tmp, sizeof(uint64_t));
#endif
if (w) /* add shifted byte from next lower word */
buff[8 * w] |= buff[8 * w - 1] << (8 - k);
Index: python-bitarray-3.7.2/bitarray/bitarray.h
===================================================================
--- python-bitarray-3.7.2.orig/bitarray/bitarray.h
+++ python-bitarray-3.7.2/bitarray/bitarray.h
@@ -273,8 +273,11 @@ popcnt_words(uint64_t *w, Py_ssize_t n)
Py_ssize_t cnt = 0;
assert(n >= 0 && ((uintptr_t) w) % 4 == 0);
- while (n--)
- cnt += popcnt_64(*w++);
+ while (n--) {
+ uintptr_t tmp;
+ memcpy(&tmp, w++, sizeof(uintptr_t));
+ cnt += popcnt_64(tmp);
+ }
return cnt;
}
With this patch, the package builds without any problems and passes the testsuite:
Successfully built bitarray-3.7.2-cp313-cp313-linux_sparc64.whl
I: pybuild plugin_pyproject:144: Unpacking wheel built for python3.13 with "installer" module
I: pybuild pybuild:334: mkdir -p /home/glaubitz/python-bitarray/python-bitarray-3.7.2/debian/buildhtml/doc && cp /home/glaubitz/python-bitarray/python-bitarray-3.7.2/README.rst /home/glaubitz/python-bitarray/python-bitarray-3.7.2/CHANGE_LOG /home/glaubitz/python-bitarray/python-bitarray-3.7.2/update_doc.py /home/glaubitz/python-bitarray/python-bitarray-3.7.2/debian/buildhtml; cp /home/glaubitz/python-bitarray/python-bitarray-3.7.2/doc/*.rst /home/glaubitz/python-bitarray/python-bitarray-3.7.2/debian/buildhtml/doc; cd /home/glaubitz/python-bitarray/python-bitarray-3.7.2/debian/buildhtml; PYTHONPATH=/home/glaubitz/python-bitarray/python-bitarray-3.7.2/.pybuild/cpython3_3.13_bitarray/build python3.13 update_doc.py; pandoc -r gfm -w html5 -o README.html -s --toc --metadata title="BitArray Documentation" README.rst; mv /home/glaubitz/python-bitarray/python-bitarray-3.7.2/debian/buildhtml/README.html /home/glaubitz/python-bitarray/python-bitarray-3.7.2
already up-to-date
dh_auto_test -a -O--buildsystem=pybuild
I: pybuild base:311: cd /home/glaubitz/python-bitarray/python-bitarray-3.7.2/.pybuild/cpython3_3.13_bitarray/build; python3.13 -c 'import bitarray; bitarray.test()'
bitarray is installed in: /home/glaubitz/python-bitarray/python-bitarray-3.7.2/.pybuild/cpython3_3.13_bitarray/build/bitarray
bitarray version: 3.7.2
sys.version: 3.13.9 (main, Oct 15 2025, 14:56:22) [GCC 15.2.0]
sys.prefix: /usr
pointer size: 64 bit
sizeof(size_t): 8
sizeof(bitarrayobject): 80
HAVE_BUILTIN_BSWAP64: 1
default bit-endianness: big
machine byte-order: big
Py_DEBUG: 0
DEBUG: 1
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 597 tests in 3.152s
OK
Thank you @glaubitz! I'm wondering if it is possible to "pre-" align the memory in way that additional memcpy's are avoided. I thought I'm already doing this in shift_r8 by calling to_aligned. I remember fixing a similar issue this way.
Sorry for the late reply!
Thank you @glaubitz! I'm wondering if it is possible to "pre-" align the memory in way that additional
memcpy's are avoided.
I know that C++ has aligned_alloc but I don't know if something similar exists in C.
I thought I'm already doing this in
shift_r8by callingto_aligned. I remember fixing a similar issue this way.
I can try to use this approach in the cases above as well.