Task watchdog got triggered

[SarahIsWeird]

Hi, my ESP32 DevKit C v2 is bootlooping when I try to use the example audio source. I was first trying to connect to a Creative Stage Air, and now I'm trying my WF-XB700 buds. I'm using PlatformIO on Windows 11, with (I suppose) the current main branch.

This is the log with verbosity 5:

ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:1044
load:0x40078000,len:10124
load:0x40080400,len:5828
entry 0x400806a8
[D][BluetoothA2DPSource.cpp:111] BluetoothA2DPSource(): BluetoothA2DPSource,
[D][BluetoothA2DPSource.cpp:154] start(): start,
[D][BluetoothA2DPSource.cpp:172] start_raw(): start_raw,
[D][BluetoothA2DPCommon.cpp:113] get_last_connection(): get_last_connection
[D][BluetoothA2DPCommon.cpp:137] get_last_connection(): => 00:00:00:00:00:00
[D][BluetoothA2DPSource.cpp:250] bt_app_work_dispatch(): bt_app_work_dispatch event 0x0, param len 0
[D][BluetoothA2DPSource.cpp:301] bt_app_task_handler(): bt_app_task_handler, sig 0x1, 0x0
[D][BluetoothA2DPSource.cpp:527] bt_av_hdl_stack_evt(): bt_av_hdl_stack_evt evt 0
[I][BluetoothA2DPCommon.cpp:255] set_scan_mode_connectable(): set_scan_mode_connectable true
[I][BluetoothA2DPSource.cpp:559] bt_av_hdl_stack_evt(): Starting device discovery...
[D][BluetoothA2DPCommon.cpp:171] clean_last_connection(): clean_last_connection
[D][BluetoothA2DPCommon.cpp:142] set_last_connection(): set_l[It_clneecothn: PS:00c0.cp0:069]0bt_D][_lup_oallA2Dk(): Dn.cove1y s art_las
_connection(): no change!
[I][BluetoothA2DPCommon.cpp:207] log_free_heap(): Available Heap: 136968
[I][BluetoothA2DPCommon.cpp:29] disconnect(): disconect a2d: 00:00:00:00:00:00
isDatBlueeoethA20x0o mor.cmp:e42]1se:n50astbc_nne_wook_d:scaech(l: tt_opn_ction
t_ap[Bltatoothn2DPCommog.0p1:10x] set_last_ctooecAion(S:uro chanpe62
_f[ee[hlup():thv2aPSoulc Hcpp:5996 00rhdlrnst av c, e[t [Bxuetoo[h]2BPComtoo.chA:2DPS lrge.rpe_59a] ): Aeai_ablr_Htape_13l80a
cess_user_state_callbacks(): process_user_state_callbacks ESP_A2D_CONNECTION_STATE_EVT: Disconnected
[E][BluetoothA2DPSource.cpp:643] bt_app_av_sm_hdlr(): bt_app_av_sm_hdlr invalid state 16711680
Guru Meditation Error: Core  0 panic'ed (LoadStoreError). Exception was unhandled.
Core 0 register dump:
PC      : 0x40090a66  PS      : 0x00060633  A0      : 0x8008f97f  A1      : 0x3ffdbda0
A2      : 0x400d15a0  A3      : 0x0000cdcd  A4      : 0xb33fffff  A5      : 0x00000001
A6      : 0x00060620  A7      : 0x0000abab  A8      : 0x0000cdcd  A9      : 0x3ffdbda0
A10     : 0x00000003  A11     : 0x00060623  A12     : 0x00060620  A13     : 0x3f401032
A14     : 0x3f401020  A15     : 0x3f400131  SAR     : 0x00000004  EXCCAUSE: 0x00000003
EXCVADDR: 0x400d15a0  LBEG    : 0x400014fd  LEND    : 0x4000150d  LCOUNT  : 0xfffffffb

ELF file SHA256: 0000000000000000

Backtrace: 0x40090a66:0x3ffdbda0 0x4008f97c:0x3ffdbdd0 0x400d1b10:0x3ffdbe10 0x400d16a9:0x3ffdbe50 0x4008fc3a:0x3ffdbe70
  #0  0x40090a66:0x3ffdbda0 in uxPortCompareSet at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:4560
      (inlined by) vPortCPUAcquireMutexIntsDisabledInternal at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portmux_impl.inc.h:86
      (inlined by) vPortCPUAcquireMutexIntsDisabled at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/portmux_impl.h:98
      (inlined by) vTaskEnterCritical at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/tasks.c:4201
  #1  0x4008f97c:0x3ffdbdd0 in xQueueGenericReceive at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/queue.c:2520
  #2  0x400d1b10:0x3ffdbe10 in BluetoothA2DPSource::bt_app_task_handler(void*) at .pio\libdeps\esp32dev\ESP32-A2DP\src/BluetoothA2DPSource.cpp:300
  #3  0x400d16a9:0x3ffdbe50 in ccall_bt_app_task_handler at .pio\libdeps\esp32dev\ESP32-A2DP\src/BluetoothA2DPSource.cpp:56 (discriminator 1)
  #4  0x4008fc3a:0x3ffdbe70 in vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c:355 (discriminator 1)

Rebooting...

I don't have an external debugger, but with some serial prints I was able to confirm that everything in the start function was executed, so the error seems to happen somewhere after executing BluetoothA2DPSource::start.

Any help would be much appreciated! (And if you need different logs, I will of course provide them.)

[pschatzmann]

For which example are you getting the error ? Did you follow https://github.com/pschatzmann/ESP32-A2DP/wiki/PlatformIO ?

[SarahIsWeird]

I was getting the error for the bt_music_sender example. I'm on vacation now so I tried it again on my MacBook, and for whatever reason it works now. I probably misconfigured something after all, even though I did follow the wiki page '^^

Sadly, the watchdog fix from 6207f7b4ff564033de807306fe00c95d5f89790c doesn't seem to work for me, the watchdog still seems to get triggered.

E (83102) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (83102) task_wdt:  - IDLE0 (CPU 0)
E (83102) task_wdt: Tasks currently running:
E (83102) task_wdt: CPU 0: btController // <-- mostly BtA2dSourceT
E (83102) task_wdt: CPU 1: IDLE1
E (83102) task_wdt: Aborting.
abort() was called at PC 0x40157ba8 on core 0

In- or decreasing the delay in the loop didn't help either. I'm still using the bt_music_sender example. (Should I create a new issue for this?)

[pschatzmann]

I can reproduce your issue. Running the sketch from Arduino is a little bit more stable. I dont have a final solution yet but I suggest to just deactivate the watchdog on core1 by calling disableCore0WDT(); in the setup

[venquessa]

In my case I modified the audiokit fft example to take from A2DP, the FFT write() is called in the A2DP data callback which runs on the BT core maxing it out so the IDLE task gets no heartbeat time and fires the watchdog timer. If anything it's letting you know you are at 100% on that core.

Disabling the watchdog timer does make the problem disappear, but I'm not sure it fixes it.

What does fix it is offloading the incoming data stream into a buffer and picking it up in loop()

This may only apply to sketches where there is nothing in the loop. "loop()" is executed (by default) on core 0, while the BT, Wifi et.al. go to Core 1.

The tricky part of off loading via buffers is that you have two cores. That means you can't get away with a laymans circular buffer setup, the concurrency issues even around incrementing pointers will kill you. The elegant solution would be to hand this off to RTOS and dispatch Work items to the loop task. I might try simple mutex's or atomics first :)

I suppose I should read the code for the copier to see if you have solved this already that way, and that's why it's in the loop.

[pschatzmann]

Are you sure that the FFT is the issue and not what you do with the FFT result? If you look at the FFT Performance would be surprosed if FFT is the issue. Did you measure the the performance for your case ?

[venquessa]

Yes. It was 1.2 millis. 1200ish uS. For a buffer length of 4096 I should have about 46 milliseconds. That was "bucketing" the bins on a fixed log scale, Then slotting the magnitudes with another fixed log scale to give 8 pixel output per band. Updating 256LED and writing them on the wire.

I tried to move to the A2DPStream and use the StreamCopy to write it to the FFT in the loop, but it produces garbage from the buffer traffic compared to the number of FFT results generated, something is very, very wrong.

[pschatzmann]

I would recommend to stick with the callback interface which is much more efficient. There you can just call write() on the fft sink object.

I tried to google the topic of log scale binning and came to the conclusion that this is complicated! And I did not find any easy implementation in C++.

The AudioTools library is supposed to be cross platform, so I did not add any functionality for task management and synchronization. i always thought that the FreeRTOS API is unfriendliy, so I started to work on this: https://github.com/pschatzmann/arduino-freertos-addons

[venquessa]

Log binning - hobby style. I don't care below 40Hz drop it. I don't care over 14kHz drop that. So I have ~14kHz range starting at 40. What do I need to multiple 40 by such that on the 32 iteration it will be approximately 14kHz. Turns out about 1.2 (aka log1.2). No more science or phsyco-acoustics involved :D On log binning the amplitude, I tried the same, but it didn't quiet work. I pinged my brother whole told me it should log3 or log6 for audio DB scale. log6 didn't work, log3 has promise and I was just narrowing it in when the flashing groups of LEDs caused by buffer corruption led me to head off down that route first.

It all comes because humans do not experience sound linearly. Neither in volume or in pitch. Most of the sound that interests us is in the lower half of our range. So scaling the "frequency band per bin" to get wider and wider at the top makes sense.

I didn't do any computing, I used a spreadsheet and stored the fixed log values as constants. No point calculating that every loop.

I'll try the event handler again. The trouble is it runs in the BT task context. It will certainly make the concurrency issues with buffers go away, but I know the margin in there is slim. Maybe after I optimised a few things, made some thing inline, told Gcc to unroll the for loops etc. I can get past the watchdog.... or... just disable it.

Sorry to do the actual binning, I simply ladder up the scale until I find the bin it goes in. In the FFT results this can even be done in a dual variable loop, incrementing the "band bucket" pointer as it passes boundaries. Presently just adding bins together, but experimenting by leaking 50% either side and also sliding averaging the buckets.... depends on performance.

I would like to do much more to the LEDs. I would like to make each pixel have a decay rate and some gravity so peaks fall. Then update all 256 repeatedly to give the display some persistence of vision and "trails", I really need my own core. I suppose I can riddle my loops with yeilds().

[venquessa]

So, after going and RTFM a bit, I managed to fix this.

Sticking to the relevant, to this ticket, part if you absolutely need to offload the FFT and it's result callback to another core, the following works for me.

`// Write data to AudioKit in callback // CORE 0 void read_data_stream(const uint8_t *data, uint32_t length) { // Expect a lazy consumer who doesn't care about losing data // And a producer much less impressed with being blocked if(xStreamBufferSpacesAvailable(xStreamBuffer) < length) { return; }

xStreamBufferSend(xStreamBuffer, data, length, 0);
}

// CORE 1 void fft_led_task(void* args) { while(1) {
size_t recv = xStreamBufferReceive(xStreamBuffer, &data, AUDIO_BUFFER_SIZE, 100); fft.write((const uint8_t*)&data, recv); } }`

Init code:

`TaskHandle_t consTask; StreamBufferHandle_t xStreamBuffer; const size_t xStreamBufferSizeBytes = 8192, xTriggerLevel = 4096;

// Create a stream buffer that can hold xStreamBufferSizeBytes bytes. The memory used to hold // both the stream buffer structure and the data in the stream buffer is // allocated dynamically. xStreamBuffer = xStreamBufferCreate( xStreamBufferSizeBytes, xTriggerLevel );

if( xStreamBuffer == NULL ) { // There was not enough heap memory space available to create the // stream buffer. Serial.println("Faled to create stream buffer, bye bye"); vTaskDelete(NULL); } xTaskCreatePinnedToCore(fft_led_task, "FFT_LED_Task",4096, NULL, 10, &consTask, 1);`

Working fine. Just need to continue my mathematically illiterate and blind fumbling to tune my analyser display.

[pschatzmann]

The issue with the watchdog is related to the ESP 1.0.6. We are currently on version 2.0.5, so please stop using this old release...