bcc
bcc copied to clipboard
iovisor
double free or corruption (!prev)
Hi, I'm hitting this issue with bcc's master branch.
double free or corruption (!prev)
Program received signal SIGABRT, Aborted.
0x00007ffff767f0f4 in __pthread_kill_implementation () from /lib64/libc.so
(gdb) bt
#0 0x00007ffff767f0f4 in __pthread_kill_implementation () from /lib64/libc.so.6
#1 0x00007ffff7625fde in raise () from /lib64/libc.so.6
#2 0x00007ffff760d942 in abort () from /lib64/libc.so.6
#3 0x00007ffff760e7a7 in __libc_message_impl.cold () from /lib64/libc.so.6
#4 0x00007ffff7689265 in malloc_printerr () from /lib64/libc.so.6
#5 0x00007ffff768b44c in _int_free_merge_chunk () from /lib64/libc.so.6
#6 0x00007ffff768dfce in free () from /lib64/libc.so.6
#7 0x00007fffd9bdf25e in std::__new_allocator<long>::deallocate (this=0x555555845b98, __p=0x2b908d, __n=356881)
at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/new_allocator.h:172
#8 std::allocator_traits<std::allocator<long> >::deallocate (__a=..., __p=0x2b908d, __n=356881) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/alloc_traits.h:513
#9 std::_Vector_base<long, std::allocator<long> >::_M_deallocate (this=0x555555845b98, __p=0x2b908d, __n=356881)
at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/stl_vector.h:389
#10 std::_Vector_base<long, std::allocator<long> >::~_Vector_base (this=0x555555845b98) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/stl_vector.h:368
#11 std::vector<long, std::allocator<long> >::~vector (this=0x555555845b98) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/stl_vector.h:738
#12 llvm::TensorSpec::~TensorSpec (this=0x555555845b70) at /usr/src/debug/llvm-19.1.5-1.fc41.x86_64/llvm/include/llvm/Analysis/TensorSpec.h:62
#13 std::_Destroy<llvm::TensorSpec> (__pointer=0x555555845b70) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/stl_construct.h:151
#14 std::_Destroy_aux<false>::__destroy<llvm::TensorSpec*> (__first=0x555555845b70, __last=0x555555846700)
at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/stl_construct.h:163
#15 std::_Destroy<llvm::TensorSpec*> (__first=<optimized out>, __last=0x555555846700) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/stl_construct.h:195
#16 std::_Destroy<llvm::TensorSpec*, llvm::TensorSpec> (__first=<optimized out>, __last=0x555555846700) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/alloc_traits.h:944
#17 std::vector<llvm::TensorSpec, std::allocator<llvm::TensorSpec> >::~vector (this=0x7fffe9297668 <llvm::FeatureMap>)
at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/stl_vector.h:735
#18 0x00007ffff7628701 in __run_exit_handlers () from /lib64/libc.so.6
#19 0x00007ffff76287ce in exit () from /lib64/libc.so.6
#20 0x00007ffff7a66538 in Py_Exit () from /lib64/libpython3.13.so.1.0
#21 0x00007ffff7a58dc9 in ?? () from /lib64/libpython3.13.so.1.0
#22 0x00007ffff7a58c19 in ?? () from /lib64/libpython3.13.so.1.0
#23 0x00007ffff785db96 in ?? () from /lib64/libpython3.13.so.1.0
#24 0x00007ffff7a58661 in ?? () from /lib64/libpython3.13.so.1.0
#25 0x00007ffff7a50c58 in Py_RunMain () from /lib64/libpython3.13.so.1.0
#26 0x00007ffff7a0a8dc in Py_BytesMain () from /lib64/libpython3.13.so.1.0
#27 0x00007ffff760f248 in __libc_start_call_main () from /lib64/libc.so.6
#28 0x00007ffff760f30b in __libc_start_main_impl () from /lib64/libc.so.6
#29 0x0000555555555095 in _start ()
My setup is Fedora 41 with llvm.x86_64 19.1.5-1.fc41, python 3.13.1 and bcc master (fea5b15d745a50). Steps to reproduce:
- Build bcc from source
- Run any tool, it doesn't even need to actually insert any ebpf. Running without permissions also causes the error. E.g:
$ python ./tools/cpudist.py 1 1
could not open bpf map: dist, error: Operation not permitted
Traceback (most recent call last):
File "/home/amorenoz/src/bcc/./tools/cpudist.py", line 248, in <module>
b = BPF(text=bpf_text, cflags=["-DMAX_PID=%d" % max_pid])
File "/usr/lib/python3.13/site-packages/bcc-0.32.0+fea5b15d-py3.13.egg/bcc/__init__.py", line 505, in __init__
raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>
double free or corruption (out)
Aborted (core dumped)
Not sure if this bug falls in bcc's side or llvm's but starting here as it's likely others will hit this issue.
Another look at this (this time with --help) yields a slightly different error message but pointing to the same area.
$ gdb -ex run --args python ./tools/cpudist.py 1 1 --help
GNU gdb (Fedora Linux) 15.2-3.fc41
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...
This GDB supports auto-downloading debuginfo from the following URLs:
<https://debuginfod.fedoraproject.org/>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Reading symbols from /home/amorenoz/.cache/debuginfod_client/59deab8a1f44002a2b5d17e12d969cfe9afdc2d8/debuginfo...
Starting program: /usr/bin/python ./tools/cpudist.py 1 1 --help
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
usage: cpudist.py [-h] [-O] [-T] [-m] [-P] [-L] [-p PID] [-I] [-e] [interval] [count]
Summarize on- and off-CPU time per task as a histogram.
positional arguments:
interval output interval, in seconds
count number of outputs
options:
-h, --help show this help message and exit
-O, --offcpu measure off-CPU time
-T, --timestamp include timestamp on output
-m, --milliseconds millisecond histogram
-P, --pids print a histogram per process ID
-L, --tids print a histogram per thread ID
-p, --pid PID trace this PID only
-I, --include-idle include CPU idle time
-e, --extension show extension summary (average/total/count)
examples:
cpudist # summarize on-CPU time as a histogram
cpudist -O # summarize off-CPU time as a histogram
cpudist 1 10 # print 1 second summaries, 10 times
cpudist -mT 1 # 1s summaries, milliseconds, and timestamps
cpudist -P # show each PID separately
cpudist -p 185 # trace PID 185 only
cpudist -I # include CPU idle time
cpudist -e # show extension summary (average/total/count)
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff768dfc1 in arena_for_chunk (ptr=0x5555556b4890) at /usr/src/debug/glibc-2.40-17.fc41.x86_64/malloc/arena.c:153
153 return chunk_main_arena (ptr) ? &main_arena : heap_for_ptr (ptr)->ar_ptr;
(gdb) bt
#0 0x00007ffff768dfc1 in arena_for_chunk (ptr=0x5555556b4890) at /usr/src/debug/glibc-2.40-17.fc41.x86_64/malloc/arena.c:153
#1 arena_for_chunk (ptr=0x5555556b4890) at /usr/src/debug/glibc-2.40-17.fc41.x86_64/malloc/arena.c:151
#2 __GI___libc_free (mem=0x5555556b48a0) at malloc.c:3397
#3 0x00007fffd9bdf275 in std::__new_allocator<char>::deallocate (this=0x5555556b5870, __p=0x7ffff77f6ac0 <main_arena> "", __n=1) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/new_allocator.h:172
#4 std::allocator_traits<std::allocator<char> >::deallocate (__a=..., __p=0x7ffff77f6ac0 <main_arena> "", __n=1) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/alloc_traits.h:513
#5 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_destroy (this=0x5555556b5870, __size=<optimized out>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/basic_string.h:294
#6 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose (this=0x5555556b5870) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/basic_string.h:288
#7 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string (this=0x5555556b5870) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/basic_string.h:809
#8 llvm::TensorSpec::~TensorSpec (this=0x5555556b5870) at /usr/src/debug/llvm-19.1.5-1.fc41.x86_64/llvm/include/llvm/Analysis/TensorSpec.h:62
#9 std::_Destroy<llvm::TensorSpec> (__pointer=0x5555556b5870) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/stl_construct.h:151
#10 std::_Destroy_aux<false>::__destroy<llvm::TensorSpec*> (__first=0x5555556b5870, __last=0x5555556b6450) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/stl_construct.h:163
#11 std::_Destroy<llvm::TensorSpec*> (__first=<optimized out>, __last=0x5555556b6450) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/stl_construct.h:195
#12 std::_Destroy<llvm::TensorSpec*, llvm::TensorSpec> (__first=<optimized out>, __last=0x5555556b6450) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/alloc_traits.h:944
#13 std::vector<llvm::TensorSpec, std::allocator<llvm::TensorSpec> >::~vector (this=0x7fffe9297668 <llvm::FeatureMap>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/stl_vector.h:735
#14 0x00007ffff7628701 in __run_exit_handlers (status=status@entry=0, listp=0x7ffff77f6680 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#15 0x00007ffff76287ce in __GI_exit (status=status@entry=0) at exit.c:138
#16 0x00007ffff7a66538 in Py_Exit (sts=0) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/pylifecycle.c:3412
#17 0x00007ffff7a58dc9 in handle_system_exit () at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/pythonrun.c:649
#18 0x00007ffff7a58c19 in _PyErr_PrintEx (tstate=0x7ffff7d3a870 <_PyRuntime+283024>, set_sys_last_vars=1) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/pythonrun.c:658
#19 0x00007ffff785db96 in PyErr_PrintEx (set_sys_last_vars=1) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/pythonrun.c:735
#20 PyErr_Print () at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/pythonrun.c:741
#21 _PyRun_SimpleFileObject (fp=fp@entry=0x555555570240, filename=filename@entry=0x7fffe9667690, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffd7f8) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/pythonrun.c:496
#22 0x00007ffff7a58661 in _PyRun_AnyFileObject (fp=fp@entry=0x555555570240, filename=filename@entry=0x7fffe9667690, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffd7f8) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Python/pythonrun.c:77
#23 0x00007ffff7a50c58 in pymain_run_file_obj (program_name=0x7fffe9636eb0, filename=0x7fffe9667690, skip_source_first_line=0) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Modules/main.c:409
#24 pymain_run_file (config=0x7ffff7d0cf68 <_PyRuntime+96392>) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Modules/main.c:428
#25 pymain_run_python (exitcode=0x7fffffffd7ec) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Modules/main.c:696
#26 Py_RunMain () at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Modules/main.c:775
#27 0x00007ffff7a0a8dc in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/python3.13-3.13.1-2.fc41.x86_64/Modules/main.c:829
#28 0x00007ffff760f248 in __libc_start_call_main (main=main@entry=0x555555555160 <main>, argc=argc@entry=5, argv=argv@entry=0x7fffffffda58) at ../sysdeps/nptl/libc_start_call_main.h:58
#29 0x00007ffff760f30b in __libc_start_main_impl (main=0x555555555160 <main>, argc=5, argv=0x7fffffffda58, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffda48) at ../csu/libc-start.c:360
#30 0x0000555555555095 in _start ()