[BUG][IPC4] ipc timed out for 0x12090000| GLB_DELETE_PIPELINE when testing multiple pipeline

[keqiaozhang]

Describe the bug Observed this issue on ADLP-NOCODEC with driver IPC4+ SOF Zephyr IPC4. CI test ID:14754 error message:

[  551.443963] kernel: snd_sof:sof_widget_free: sof-audio-pci-intel-tgl 0000:00:1f.3: widget pipeline.10 freed
[  551.443963] kernel: snd_sof:sof_widget_free: sof-audio-pci-intel-tgl 0000:00:1f.3: widget copier.SSP.10.1 freed
[  551.443965] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx      : 0x12090000|0x0: GLB_DELETE_PIPELINE
[  551.708418] kernel: snd_sof:sof_ipc4_set_pipeline_state: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc4 set pipeline 4 state 3
[  551.950957] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc timed out for 0x12090000|0x0
[  551.950993] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: failed to free pipeline widget pipeline.9
[  551.951008] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: Failed to free connected widgets
[  551.951037] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at snd_soc_pcm_component_hw_free on 0000:00:1f.3: -110

To Reproduce almost 100%

Reproduction Rate TPLG=/lib/firmware/intel/avs-tplg/cavs-tgl-nocodec.tplg MODEL=ADLP_RVP_NOCODEC_IPC4 ~/sof-test/test-case/multiple-pipeline.sh -f p -c 20 -l 50

Environment Kernel Branch: topic/sof-dev Kernel Commit: d8dba897d903

SOF Branch: main SOF Commit: 44a5200c8762

Zephyr Commit: zephyr-v3.1.0-3042-g8e55e59c5917

dmesg.txt

[lgirdwood]

IPC timeouts are P1. @juimonen are you able to get any mtrace out to help with this ?

[juimonen]

@lgirdwood so upxtreme has currently only 1 ssp, so cant test multipipeline case in that. Need to find some device to work with.

[keqiaozhang]

@lgirdwood so upxtreme has currently only 1 ssp, so cant test multipipeline case in that. Need to find some device to work with.

I checked with vendor about how many SSP we can use for our validation on TGL UP xtreme i11 board and can we enable them all, the answer is:

Unfortunately it is not possible as the SSP0 is only available when the CPLD is enabled.
The AAM pins are routed directly from the SoC when the CPLD is disabled and no I2S signals
are available on our TGL platform due to TGL SoC pin configuration/limitations.

[keqiaozhang]

In today's inner test, the IPC ID is changed to 0x3200000c:

[ 2102.797626] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx      : 0x12030000|0x0: GLB_DELETE_PIPELINE
[ 2102.797827] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx reply: 0x3200000c|0x0: GLB_DELETE_PIPELINE
[ 2102.797833] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: FW reported error: 12 - Required resource is in invalid state
[ 2102.797865] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc error for msg 0x12030000|0x0
[ 2102.797869] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: failed to free pipeline widget pipeline.3
[ 2102.797884] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx      : 0x12040000|0x0: GLB_DELETE_PIPELINE
[ 2102.798076] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx reply: 0x3200000c|0x0: GLB_DELETE_PIPELINE
[ 2102.798078] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: FW reported error: 12 - Required resource is in invalid state
[ 2102.798106] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc error for msg 0x12040000|0x0
[ 2102.798109] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: failed to free pipeline widget pipeline.4
[ 2102.798112] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: Failed to free connected widgets
[ 2102.798120] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at snd_soc_pcm_component_hw_free on 0000:00:1f.3: -22
[ 2102.798141] kernel: snd_sof:sof_pcm_close: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: close stream 1 dir 0

https://sof-ci.sh.intel.com/#/result/planresultdetail/15002?model=TGLU_RVP_NOCODEC_IPC4ZPH&testcase=check-pause-resume-playback-100

[kv2019i]

This issue no longer occuring in daily test plan on these platforms. Closing the bug.

[lyakh]

Re-opening because I can reproduce very similar timeout errors on UP-Xtreme i11 with nocodec with two loops: one with aplay and another one with arecord. But even more often the IPC that fails is GLB_SET_PIPELINE_STATE

[keqiaozhang]

This issue is not reproducible in CI. Closing this bug.