Segfaults when compressing files with blosc2

[adam-sikora]

We are encountering occasional segfaults in our docker containers while compressing data through blosc2 library. Most of the runs everything works just fine but multiple times a day the container just crashes on segfault when compressing data. We use just one thread for compression (blosc2.set_nthreads(1)) and we call the following function to compress data blosc2.pack_array2(x, cparams={"codec": blosc2.Codec.ZSTD, "clevel": 4}) where x is non-empty 2D numpy array.

Our containers are based on ubuntu:22.04 and they are running on Google Kubernetes Engine on c2-standard-16 (Intel Cascade Lake) machines.

Affected Blosc2 versions: 2.3.1 - 2.5.1 (newest version as of time of writing)

Unfortunately I was unable to reproduce this locally so I don't have a coredump file.

I have checked ~20 occurrences of the crash and it always occurs on the first invocation of blosc2.pack_array2. If the first call to this function within the container passes fine then the crash does never occur in subsequent calls.

[FrancescAlted]

That's weird, and a pity that you cannot reproduce it in your machine. Guessing what's going on Google Kubernetes Engine is hard. Having said that, it would be nice to isolate the machines that make things go wild. Could you please run:

gcc -msse3 -dM -E - < /dev/null | egrep "SSE|AVX" | sort
python -c"import cpuinfo; print(cpuinfo.get_cpu_info()['flags'])"

prior to execute your jobs in machines and guess if this is happening for boxes that do not support AVX2? You said that c2-standard-16 is always Cascade Lake, which has support for it indeed, but just curious.

[adam-sikora]

Here is the output from the container that ran without issues: gcc -msse3 -dM -E - < /dev/null | egrep "SSE|AVX" | sort

#define __MMX_WITH_SSE__ 1
#define __SSE2_MATH__ 1
#define __SSE2__ 1
#define __SSE3__ 1
#define __SSE_MATH__ 1
#define __SSE__ 1

python -c"import cpuinfo; print(cpuinfo.get_cpu_info()['flags'])"

['3dnowprefetch', 'abm', 'adx', 'aes', 'apic', 'arat', 'arch_capabilities', 'avx', 'avx2', 'avx512_vnni', 'avx512bw', 'avx512cd', 'avx512dq', 'avx512f', 'avx512vl', 'avx512vnni', 'bmi1', 'bmi2', 'clflush', 'clflushopt', 'clwb', 'cmov', 'constant_tsc', 'cpuid', 'cx16', 'cx8', 'de', 'erms', 'f16c', 'fma', 'fpu', 'fsgsbase', 'fxsr', 'hle', 'ht', 'hypervisor', 'ibpb', 'ibrs', 'ibrs_enhanced', 'invpcid', 'invpcid_single', 'lahf_lm', 'lm', 'mca', 'mce', 'md_clear', 'mmx', 'movbe', 'mpx', 'msr', 'mtrr', 'nonstop_tsc', 'nopl', 'nx', 'osxsave', 'pae', 'pat', 'pcid', 'pclmulqdq', 'pdpe1gb', 'pge', 'pni', 'popcnt', 'pse', 'pse36', 'rdrand', 'rdrnd', 'rdseed', 'rdtscp', 'rep_good', 'rtm', 'sep', 'smap', 'smep', 'ss', 'ssbd', 'sse', 'sse2', 'sse4_1', 'sse4_2', 'ssse3', 'stibp', 'syscall', 'tsc', 'tsc_adjust', 'tsc_known_freq', 'vme', 'x2apic', 'xgetbv1', 'xsave', 'xsavec', 'xsaveopt', 'xsaves', 'xtopology']

I will post an update once we hit the issue again but I expect the CPU flags to be the same.

[adam-sikora]

I was able to hit the issue in our containers again and as predicted the CPU flags are the same as the ones posted in previous comment

[adam-sikora]

I was able to get hold of a coredump. This is the backtrace:

(gdb) backtrace
#0  0x0000000000000000 in ?? ()
#1  0x00007f5f031e027a in pipeline_forward ()
   from /usr/local/lib/python3.10/dist-packages/blosc2/blosc2_ext.cpython-310-x86_64-linux-gnu.so
#2  0x00007f5f031e0700 in blosc_c () from /usr/local/lib/python3.10/dist-packages/blosc2/blosc2_ext.cpython-310-x86_64-linux-gnu.so
#3  0x00007f5f031e9524 in t_blosc_do_job ()
   from /usr/local/lib/python3.10/dist-packages/blosc2/blosc2_ext.cpython-310-x86_64-linux-gnu.so
#4  0x00007f5f031e99e2 in t_blosc () from /usr/local/lib/python3.10/dist-packages/blosc2/blosc2_ext.cpython-310-x86_64-linux-gnu.so
#5  0x00007f5f7902dac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f5f790bea04 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Is this sufficient to fix the issue or do you need more data?

EDIT: Python callstack:

File "/usr/local/lib/python3.10/dist-packages/blosc2/schunk.py", line 237 in __init__
File "/usr/local/lib/python3.10/dist-packages/blosc2/core.py", line 617 in pack_tensor
File "/usr/local/lib/python3.10/dist-packages/blosc2/core.py", line 445 in pack_array2
...

[FrancescAlted]

Yes, this is informative. It looks like the issue is in the shuffle filter (called from pipeline_forward), which is enabled by default. However, shuffle has been declared apt for production inside Blosc for more than 13 years now, and we never detected an issue since then, so I am a bit surprised here...

Again, the best would be to have a small program that can reproduce the issue.

[FrancescAlted]

I recently fixed an issue that caused a division by zero on machines with 1 single core. It might be that sometimes kubernetes sent a job to a machine with just a single CPU? That could have triggered the issue.

I've just released python-blosc2 2.6.2, with the fix in.

[adam-sikora]

No we are for sure running on multi CPU machines as the deployments request 4 CPUs so this will most likely not be related.

I'm still not able to reproduce the issue locally unfortunately. Will post an update if I find any more information

[adam-sikora]

I can confirm that the crashes are still happening on the newest 2.6.2 version.

Despite my efforts I still don't have an easy way to reproduce the issue.

Currently I'm trying to obtain a coredump with blosc built with RelWithDebInfo/Debug configuration. Hopefully this way we can get sufficient information to fix the issue.

[adam-sikora]

Building the blosc2 Python package from source seems to resolve the issues. We have deployed this workaround to production and we have not seen a single segfault after one week.

This means that the crashes only appear when we use the prebuilt pip wheel. Do you have insight into why this could cause the segfaults?

[FrancescAlted]

Thanks for sharing your experience. It occurs to me that this might be some bug that has been fixed in recent compilers; with that, perhaps building wheels with more modern base systems would help. Could you try building your own wheels with manylinux_2_28 and see how it goes? Doing that should be as easy as adding the next lines to pyproject.yaml and create the wheel as usual.

[tool.cibuildwheel]
...
manylinux-x86_64-image = "manylinux_2_28"
manylinux-aarch64-image = "manylinux_2_28"

[adam-sikora]

Could you perhaps supply such wheel? This way I don't have to test if the crashes still occur in the wheel built by me from master to rule out other possible causes for disappearance of the crashes.

Anyway the testing will again take some time as I first have to rebuild all our containers and then wait for the crashes to appear again in our devel environment, which does not see as much traffic as the production environment but the crashes happen there as well from time to time when running our integration test suite.

[FrancescAlted]

We recently have released 2.7.1 fixing SIGKILL issues. Could you try the new one?

[adam-sikora]

From the first tests it seems that the issue is fixed in 2.7.1. I will post a final confirmation in few weeks once we deploy it to production and have some time to verify that the segfaults no longer happen.

[adam-sikora]

I can confirm that the issue was fixed and we did not observe any crashes after update

[FrancescAlted]

Great! Thanks for your feedback!