sof icon indicating copy to clipboard operation
sof copied to clipboard

ipc4: implement mtrace logging support

Open kv2019i opened this issue 3 years ago • 1 comments

Set of patches to implement mtrace support in SOF firmware.

Requires Zephyr and Zephyr PR https://github.com/zephyrproject-rtos/zephyr/pull/49147 to work.

V1 update:

  • otherwise ready to, but some hacks in this version related to sending notifies when a IPC message is under processing -- a crude hack is this patch to solve the issue, but we probably need something a bit cleaner

kv2019i avatar Aug 18 '22 07:08 kv2019i

For the record, I'm looking for something hard coded and simple to get data out as the first merge point. We can then improve with dynamic config/tuning in subsequent updates.

lgirdwood avatar Aug 18 '22 12:08 lgirdwood

V3 update:

  • major clean up, this should be now ready for @lgirdwood 's criteria of a minimal working set
  • keeping still as draft as the Zephyr PR needs to be merged first. I can then undraft this and submit this together with a manifest update
  • the aging timer task is a bit ugly still, but I had major issues with other options I tried - trying to use schedule_task from logging hook results in recursion and non-functional FW - k_timer and k_work deliver callbacks in irq context (and possibly on non-zero core) and I can't figure out how to send the IPC notifies safely from this context

kv2019i avatar Aug 23 '22 14:08 kv2019i

V4 update:

  • implemented the worker thread using native interfaces -- a bit more code, but this is more efficient as we don't need to run anything extra in the 1ms ll cadence
  • added overlay changes to make this the default for all IPC4 platforms
  • fixed a few bugs w.r.t. draining logs at end when streaming is stopped
  • still keeping as draft as the Zephyr side PR is still in review

kv2019i avatar Aug 24 '22 16:08 kv2019i

All, the important thing here is that we need some data out as the first step. Today we have no logging. We can improve our logging infra incrementally (and we can track this with FEATURES).

lgirdwood avatar Aug 26 '22 10:08 lgirdwood

V5 update:

  • improved the SOF EDF adaptation to Zephyr a bit and implemented the mtrace work thread logic using edf_schedule.h (this is basicly a k_thread pinned on a thread... and it's a cost we are already taking so why not piggyback on this)
  • bugfix to handle runtime suspend cases to mtrace-reader.py
  • still keeping as draft as the Zephyr part is not merged

kv2019i avatar Aug 26 '22 17:08 kv2019i

V6 update:

  • not included a manifest update to bring in the Zephyr dependencies
  • a few bugfixes to the EDF adaptation for recurring EDF tasks
  • added a more clear TODO for multicore; I spent a lot of time trying to use notifier.h to wake up a task from one core to another, but kept hitting with weird issues... this version now explicitly optimized for single-core usege. it will work wiht multicore, but one may need to use a more aggressive aging timer not to lose logs

kv2019i avatar Aug 29 '22 16:08 kv2019i

Zephyr builds errors are due to old SDK used in CI. 0.15 SDK is needed to build (or XCC).

kv2019i avatar Aug 29 '22 16:08 kv2019i

@keqiaozhang @fredoh9 @marc-hb can we update the Zephyr SDK to align with the version used by Zephyr. Thanks

lgirdwood avatar Aug 29 '22 19:08 lgirdwood

V7 update:

  • updated github workflows to use a specific zephyr docker-image that containts 0.15 SDK
  • there is one IPC4 build failure left but that is a compiler warning handled via https://github.com/thesofproject/sof/pull/6211
  • no change to code otherwise

kv2019i avatar Aug 29 '22 19:08 kv2019i

@keqiaozhang @fredoh9 @marc-hb can we update the Zephyr SDK to align with the version used by Zephyr. Thanks

Pls ignore, @kv2019i has updated in the PR.

lgirdwood avatar Aug 29 '22 19:08 lgirdwood

V8 update:

kv2019i avatar Aug 29 '22 20:08 kv2019i

V9 update:

  • addressed @abonislawski 's revies and most notably reviewed mtrace.[ch] to logging.[ch] as the same IPC can be used for other logging backends
  • one bugfix to disabling logs that was causing IPC timeouts in testing
  • kept the SDK update... the "latest" tag still points to 0.14 SDK

kv2019i avatar Aug 30 '22 12:08 kv2019i

Zephyr IPC4 build fix merged.

lgirdwood avatar Aug 31 '22 09:08 lgirdwood