[BUG] Volume DSP Panic on IPC response timeout, DSP Panic notification

[wszypelt]

Describe the bug

ERROR:Exception DspPanicError occurred at 'C:\work\tools\cavs_scripts-py\utilities\ipc_driver.py':171 DSP Panic on IPC response timeout! 0x00000005 0x0DEAD006 0x00004000 ERROR:Exception DspPanicError occurred at 'C:\work\tools\cavs_scripts-py\utilities\ipc_driver.py':171 DSP Panic notification received: SOF_IPC_PANIC_EXCEPTION 0x00000005 0x0DEAD006 0x00004000

Bug occurs with a playback iteration greater than 1

Bug was found in: TestVolumeSimple8000Hz16b16b1ch TestVolumeSimple8000Hz24b32b4ch TestVolumeSimple48000Hz32b32b8ch TestVolumeChangeInMiddle8000Hz32b32b4ch TestVolumeChangeInMiddle48000Hz32b32b4ch TestVolumeMultipleChanges8000Hz24b32b2ch TestVolumeMultipleChanges48000Hz24b32b2ch TestVolumeMute8000Hz32b32b8ch TestVolumeMute48000Hz32b32b8ch TestVolumeInitialRampSimple8000Hz24b32b4ch TestVolumeInitialRampSimple48000Hz32b32b2ch TestDiffVolumeChangeInChannels8000Hz24b32b4ch TestDiffVolumeChangeInChannels48000Hz32b32b4ch

Topology

TestVolumeSimple

                +--------------------------------+
    +------+    | +---+  +- -+  +---+  +-------+ |
    | Host |----->|Buf|->|Vol|->|Buf|->|SSP Dai|---+
    +------+    | +---+  +-- +  +---+  +-------+ | |
                +--------------------------------+ |
                                                   |
                +------------------+               |
    +------+    | +---+  +-------+ |               |
    | Host |<-----|Buf|<-|SSP Dai|<----------------+
    +------+    | +---+  +-------+ |
                +------------------+

Description: Simultaneous playback and capture on external loopback (same SSP) with signal amplitude verification. Playback pipeline contains volume component. Volume gain is explicitly set to 1 before playback starts.

To Reproduce Run tests with diagnostic driver with argument: --playback_iterations=2

03_00_TestVolumeSimple8000Hz16b16b1ch

Reproduction Rate 100%

Environment

1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
   • SOF BRANCH: main HASH: ea72e3e97e10acb27d02c3692b006c4f66fabfe4
2. Name of the platform(s) on which the bug is observed.
   • Platform: TGL RVP

the last working and not working commit: ebc222dab56868ea3cb0d5abec5057990eae6bb9

Logs DSPPanicVolume.zip

[lgirdwood]

@ranj063 module adapter related ?

[kfrydryx]

Still reproducing. Bisect points to module adapter - first bad commit f7663de1f41308b80bac4618b5835fcda1ff8fac.

[lgirdwood]

@wszypelt @kfrydryx is this only reproducing with the loopback ? What happens if gain is NOT set prior to test. The DEAD code is an assert(). @lyakh can you take a look at this, thanks !

[lyakh]

@wszypelt can this still be reproduced? But this test isn't in the standard CI, so we don't see it in PR testing?

[wszypelt]

@lyakh @kfrydryx @lgirdwood Issue still reproducing, with playback iteration > 1. In CI we test all tests with playback_iteration = 1. Therefore, it does not show this issue. We perform stress tests every two weeks.

[lgirdwood]

@lyakh @kfrydryx @lgirdwood Issue still reproducing, with playback iteration > 1. In CI we test all tests with playback_iteration = 1. Therefore, it does not show this issue. We perform stress tests every two weeks.

Ok, so the Linux CI will test multiple times, but I dont think it uses the loopback mode above. @fredoh9 can you confirm if we have a similar loopback test in the Jenkins CI ?