drowe67
Enabling libasan causes failures during tests
@barjac reported an AddressSanitizer error during freedv-gui build:
[100%] Built target freedv_datac0c1_rx
make[1]: Leaving directory '/home/baz/build/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/build_linux'
/usr/bin/cmake -E cmake_progress_start /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/build_linux/CMakeFiles 0
+ cd src
+ export LD_LIBRARY_PATH=/home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/LPCNet/build_linux/src
+ LD_LIBRARY_PATH=/home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/LPCNet/build_linux/src
+ ./freedv_tx 2020 /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/LPCNet/wav/wia.wav -
+ ./freedv_rx 2020 - /dev/null
=================================================================
==36243==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffefa65e990 at pc 0x000000402ffb bp 0x7ffefa6593d0 sp 0x7ffefa6593c8
READ of size 8 at 0x7ffefa65e990 thread T0
#0 0x402ffa in my_get_next_tx_char /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_tx.c:46
#1 0x7f66ff6b3c02 in freedv_comptx_2020 /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_2020.c:174
#2 0x7f66ff6a575c in freedv_comptx /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_api.c:389
#3 0x7f66ff6a5ea5 in freedv_tx /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_api.c:329
#4 0x40284d in main /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_tx.c:205
#5 0x7f66ff47c736 in __libc_start_call_main (/lib64/libc.so.6+0x23736)
#6 0x7f66ff47c7f4 in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x237f4)
#7 0x402ea0 in _start (/home/baz/build/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/build_linux/src/freedv_tx+0x402ea0)
Address 0x7ffefa65e990 is located in stack of thread T0 at offset 240 in frame
#0 0x40231f in main /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_tx.c:59
This frame has 3 object(s):
[32, 36) 'opt_idx' (line 97)
[48, 128) 'f2020' (line 66)
[160, 256) 'my_cb_state' (line 187) <== Memory access at offset 240 is inside this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-scope /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_tx.c:46 in my_get_next_tx_char
Shadow bytes around the buggy address:
0x10005f4c3ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10005f4c3cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10005f4c3d00: cb cb cb cb 00 00 00 00 00 00 00 00 00 00 00 00
0x10005f4c3d10: 00 00 00 00 f1 f1 f1 f1 04 f2 00 00 00 00 00 00
0x10005f4c3d20: 00 00 00 00 f2 f2 f2 f2 f8 f8 f8 f8 f8 f8 f8 f8
=>0x10005f4c3d30: f8 f8[f8]f8 f3 f3 f3 f3 00 00 00 00 00 00 00 00
0x10005f4c3d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10005f4c3d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10005f4c3d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10005f4c3d70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10005f4c3d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==36243==ABORTING
frames decoded: 0 output speech samples: 0
+ cd /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864
Additionally, it appears a large number of ctests fail with AddressSanitizer enabled. Building with the following on an aarch64 Ubuntu VM:
$ CFLAGS=-fsanitize=address LDFLAGS=-fsanitize=address cmake -DLPCNET_BUILD_DIR=../../LPCNet/build_linux -DUNITTEST=1 ..
$ make -j6
$ ASAN_OPTIONS=detect_odr_violation=0 make test
....
61% tests passed, 60 tests failed out of 155
Total Test time (real) = 252.27 sec
The following tests FAILED:
1 - test_clang_format (Failed)
13 - test_ch_papr (Failed)
19 - test_COHPSK_modem_octave_port (Failed)
20 - test_COHPSK_modem_AWGN_BER (Failed)
21 - test_COHPSK_modem_freq_offset (Failed)
23 - test_OFDM_modem_octave_port (Failed)
24 - test_OFDM_modem_octave_port_Nc_31 (Failed)
34 - test_OFDM_modem_700D (Failed)
35 - test_OFDM_modem_700D_ldpc (Failed)
36 - test_OFDM_modem_2020_ldpc (Failed)
37 - test_OFDM_modem_AWGN_BER (Failed)
38 - test_OFDM_modem_fading_BER (Failed)
39 - test_OFDM_modem_phase_est_bw (Failed)
41 - test_OFDM_modem_time_sync_2020 (Failed)
42 - test_OFDM_modem_700E_AWGN (Failed)
43 - test_OFDM_modem_2020B_AWGN (Failed)
47 - test_OFDM_modem_datac1 (Failed)
51 - test_OFDM_modem_datac13_octave (Failed)
52 - test_OFDM_modem_datac4_ldpc_burst (Failed)
53 - test_OFDM_modem_datac13_ldpc_burst (Failed)
66 - test_freedv_api_1600 (Failed)
68 - test_freedv_api_700D_backwards_compatability (Failed)
69 - test_freedv_api_700D_speech (Failed)
71 - test_freedv_api_700D_AWGN_BER (Failed)
72 - test_freedv_api_700D_AWGN_BER_USECOMPLEX (Failed)
76 - test_freedv_api_2020_to_ofdm_demod (Failed)
77 - test_freedv_api_2020_from_ofdm_mod (Failed)
78 - test_freedv_api_2020_awgn (Failed)
79 - test_freedv_api_2020B_mpp (Failed)
80 - test_freedv_api_2400A (Failed)
81 - test_freedv_api_2400B (Failed)
82 - test_freedv_api_800XA (Failed)
83 - test_freedv_api_rawdata_800XA (Failed)
84 - test_freedv_api_rawdata_2400A (Failed)
85 - test_freedv_api_rawdata_2400B (Failed)
86 - test_peak_levels (Failed)
87 - test_peak_levels_lpcnet (Failed)
98 - test_freedv_reliable_text_ideal_2020 (Failed)
100 - test_freedv_reliable_text_fade_2020 (Failed)
101 - test_memory_leak_FreeDV_1600_tx (Failed)
102 - test_memory_leak_FreeDV_1600_rx (Failed)
103 - test_memory_leak_FreeDV_700D_tx (Failed)
104 - test_memory_leak_FreeDV_700D_rx (Failed)
105 - test_memory_leak_FreeDV_700C_tx (Failed)
106 - test_memory_leak_FreeDV_700C_rx (Failed)
107 - test_memory_leak_FreeDV_FSK_LDPC_tx (Failed)
108 - test_memory_leak_FreeDV_DATAC0_tx (Failed)
109 - test_memory_leak_FreeDV_DATAC1_tx (Failed)
110 - test_memory_leak_FreeDV_DATAC3_tx (Failed)
111 - test_memory_leak_FreeDV_DATAC4_tx (Failed)
112 - test_memory_leak_FreeDV_DATAC13_tx (Failed)
113 - test_memory_leak_FreeDV_700E_tx (Failed)
114 - test_memory_leak_FreeDV_2020_tx (Failed)
115 - test_memory_leak_FreeDV_2020_rx (Failed)
116 - test_memory_leak_FreeDV_2020B_tx (Failed)
117 - test_memory_leak_FreeDV_2020B_rx (Failed)
125 - test_vq_mbest (Failed)
145 - test_freedv_data_raw_ofdm_datac4_burst_file (Failed)
146 - test_freedv_data_raw_ofdm_datac13_burst_file (Failed)
153 - test_demo_700d_python (Failed)
Errors while running CTest
Output from these tests are in: /home/parallels/freedv-gui/codec2/build_linux/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make: *** [Makefile:91: test] Error 8
(More tests were failing without disabling the ODR detection. I disabled above since it seemed like a false alarm.)
Example of one of the ctest failures:
test 69
Start 69: test_freedv_api_700D_speech
69: Test command: /usr/bin/sh "-c" "cd /home/parallels/freedv-gui/codec2/build_linux/src;
./freedv_tx 700D ../../raw/ve9qrp_10s.raw - |
./ch - - --No -20 |
./freedv_rx 700D - /dev/null --squelch -2 -vv"
69: Test timeout computed to be: 1500
69: ch: Fs: 8000 NodB: -20.00 foff: 0.00 Hz fading: 0 nhfdelay: 0 clip: 32767.00 ssbfilt: 1 complexout: 0
69: =================================================================
69: ==49821==ERROR: AddressSanitizer: stack-use-after-scope on address 0xffffce666d80 at pc 0xaaaab4232d98 bp 0xffffce662bb0 sp 0xffffce662bc0
69: READ of size 8 at 0xffffce666d80 thread T0
69: #0 0xaaaab4232d94 in my_get_next_tx_char /home/parallels/freedv-gui/codec2/src/freedv_tx.c:46
69: #1 0xffff841c6a40 in freedv_comptx_ofdm /home/parallels/freedv-gui/codec2/src/freedv_700.c:266
69: #2 0xffff841bd2a4 in freedv_comptx /home/parallels/freedv-gui/codec2/src/freedv_api.c:375
69: #3 0xffff841bd984 in freedv_tx /home/parallels/freedv-gui/codec2/src/freedv_api.c:329
69: #4 0xaaaab42328f0 in main /home/parallels/freedv-gui/codec2/src/freedv_tx.c:205
69: #5 0xffff83f273f8 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
69: #6 0xffff83f274c8 in __libc_start_main_impl ../csu/libc-start.c:392
69: #7 0xaaaab4232c2c in _start (/home/parallels/freedv-gui/codec2/build_linux/src/freedv_tx+0x2c2c)
69:
69: Address 0xffffce666d80 is located in stack of thread T0 at offset 128 in frame
69: #0 0xaaaab42321ac in main /home/parallels/freedv-gui/codec2/src/freedv_tx.c:59
69:
69: This frame has 3 object(s):
69: [32, 36) 'opt_idx' (line 97)
69: [48, 144) 'my_cb_state' (line 187) <== Memory access at offset 128 is inside this variable
69: [176, 256) 'f2020' (line 66)
69: HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
69: (longjmp and C++ exceptions *are* supported)
69: SUMMARY: AddressSanitizer: stack-use-after-scope /home/parallels/freedv-gui/codec2/src/freedv_tx.c:46 in my_get_next_tx_char
69: Shadow bytes around the buggy address:
69: 0x200ff9cccd60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69: 0x200ff9cccd70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69: 0x200ff9cccd80: cb cb cb cb 00 00 00 00 00 00 00 00 00 00 00 00
69: 0x200ff9cccd90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69: 0x200ff9cccda0: f1 f1 f1 f1 04 f2 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
69: =>0x200ff9cccdb0:[f8]f8 f2 f2 f2 f2 00 00 00 00 00 00 00 00 00 00
69: 0x200ff9cccdc0: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
69: 0x200ff9cccdd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69: 0x200ff9cccde0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69: 0x200ff9cccdf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69: 0x200ff9ccce00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
69: Shadow byte legend (one shadow byte represents 8 application bytes):
69: Addressable: 00
69: Partially addressable: 01 02 03 04 05 06 07
69: Heap left redzone: fa
69: Freed heap region: fd
69: Stack left redzone: f1
69: Stack mid redzone: f2
69: Stack right redzone: f3
69: Stack after return: f5
69: Stack use after scope: f8
69: Global redzone: f9
69: Global init order: f6
69: Poisoned by user: f7
69: Container overflow: fc
69: Array cookie: ac
69: Intra object redzone: bb
69: ASan internal: fe
69: Left alloca redzone: ca
69: Right alloca redzone: cb
69: Shadow gap: cc
69: ==49821==ABORTING
69: ch: SNR3k(dB): nan C/No....: nan
69: ch: peak.....: 0.00 RMS.....: nan CPAPR.....: nan
69: ch: Nsamples.: 0 clipped.: nan% OutClipped: nan%
69: frames decoded: 0 output speech samples: 0
69:
69: =================================================================
69: ==49822==ERROR: LeakSanitizer: detected memory leaks
69:
69: Direct leak of 9 byte(s) in 1 object(s) allocated from:
69: #0 0xffffb5da8e30 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:454
69: #1 0xaaaac0d02888 in main /home/parallels/freedv-gui/codec2/src/ch.c:152
69: #2 0xffffb59473f8 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
69: #3 0xffffb59474c8 in __libc_start_main_impl ../csu/libc-start.c:392
69: #4 0xaaaac0d0442c in _start (/home/parallels/freedv-gui/codec2/build_linux/src/ch+0x442c)
69:
69: SUMMARY: AddressSanitizer: 9 byte(s) leaked in 1 allocation(s).
1/1 Test #69: test_freedv_api_700D_speech ......***Failed Required regular expression not found. Regex=[frames decoded: 62 output speech samples: 7
] 1.11 sec
A quick look seems to indicate that it's due to the following definition for my_cb_state (in both freedv_rx and freedv_tx):
} else {
/* set up callback for txt msg chars */
struct my_callback_state my_cb_state;
sprintf(my_cb_state.tx_str, "cq cq cq hello world\r");
my_cb_state.ptx_str = my_cb_state.tx_str;
my_cb_state.calls = 0;
freedv_set_callback_txt(freedv, NULL, &my_get_next_tx_char, &my_cb_state);
}
This definition should be moved to the top of main() to eliminate this error.
Thanks @tmiw. If you think you have a quick fix and don't mind working on it pls feel free to go ahead and raise a PR.
I created https://github.com/drowe67/codec2/pull/45 with what I was able to get to tonight. There's still something like 14 ctest failures with libasan on, but I suspect at least a couple are due to configuration issues on my end.
@tmiw - did any of the issues detected by AddressSanitizer cause any actual, real world issues, when running say freedv-gui? The changes in https://github.com/drowe67/codec2/pull/45 are minor tweaks mainly in test code for soon to be deprecated modes. All this code was running fine before this tool was applied.
We already have a suite of valgrind tests that I guess traps most memory issues.
@tmiw - did any of the issues detected by AddressSanitizer cause any actual, real world issues, when running say freedv-gui? The changes in #45 are minor tweaks mainly in test code for soon to be deprecated modes. All this code was running fine before this tool was applied.
We already have a suite of
valgrindtests that I guess traps most memory issues.
This originally kicked off because @barjac couldn't build with AddressSanitizer while attempting to debug a freedv-gui issue. Fixing that of course ended up kicking off additional issues (for example, the behavior apparently varying wildly depending on what locale freedv-gui is set to). Also, I'm fairly sure he was able to duplicate the crashes in freedv-gui without AddressSanitizer enabled at one point, too.
For reference, the most recent crashes seemed to be happening while trying to decode 800XA:
==261635==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000185637 at pc 0x7f503ae47971 bp 0x7f501a342f50 sp 0x7f501a342700
WRITE of size 8 at 0x602000185637 thread T23
#0 0x7f503ae47970 in __interceptor_memset (/lib64/libasan.so.8+0x47970)
#1 0x7f503acbc176 in fvhff_extract_frame_voice /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_vhf_framing.c:556
#2 0x7f503acbef19 in fvhff_extract_frame /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_vhf_framing.c:738
#3 0x7f503acbef19 in fvhff_deframe_bits /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_vhf_framing.c:855
#4 0x7f503acbaf18 in freedv_comprx_fsk /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_fsk.c:671
#5 0x7f503aca9d8a in freedv_comprx /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/codec2/src/freedv_api.c:806
#6 0x4a45f8 in FreeDVReceiveStep::execute(std::shared_ptr<short>, int, int*) /home/baz/BLD/BLD_MgaX_FREEDV/freedv-git/SOURCES/freedv-1.9.9-202403221300-3e864/src/pipeline/FreeDVReceiveStep.cpp:100
(and earlier, were happening when decoding 2020 too)
There are also security implications, too. While admittedly unlikely, I imagine someone could come up with an OTA signal (or even just a WAV file that they convinced someone to play back in freedv-gui) that triggered one of the buffer overflows fixed by this PR and caused them to gain control of the user's system or something.
Anyway, that's a long way of saying that these modes should be fixed as long as the modes are still being used by something (e.g. freedv-gui). :)
@tmiw - I'm not convinced this is an issue we should be addressing at this time.
We made a decision at PLT that non-critical bugs - especially in deprecated modes - would not be addressed at this time. Fixing ctests in deprecated code (that pass just fine) is not where we should working right now.
As codec 2 lead I'm making the call to pause any further libcodec2 work in this area - if you'd like to discuss further please raise at PLT.