ubuntu-asahi icon indicating copy to clipboard operation
ubuntu-asahi copied to clipboard

Getting Kernel panic "Asynchronous SError Interrupt" after upgrading to 6.12.0-1002.2

Open Rad0van opened this issue 1 year ago • 33 comments

Getting kernel panic on boot after upgrading to 6.12.0-1002.2. However this happens also if 6.10.0-1001.1 is chosen and even in recovery mode. It is 24.10 KDE desktop upgraded from 24.04. Clueless what to do here. MacBook Pro Max M2.

Rad0van avatar Jan 19 '25 23:01 Rad0van

Sorry for that. Can you share the crash log? The more info we have the easier it will be to track down.

tobhe avatar Jan 20 '25 00:01 tobhe

Sorry for that. Can you share the crash log? The more info we have the easier it will be to track down.

Gladly but I do not know how. It starts booting and then crashes to that pink screen. Is there some key sequence to show the boot log?

Rad0van avatar Jan 20 '25 07:01 Rad0van

I encountered the same issue here on a M2 MBP after updating Ubuntu. I do not know how to access the crash logs either, it shows a screen with a ASCII linux penguin at the top left and then says something along the lines of "kernel panic" and "reboot your computer". This seems to be the new DRM panic screen: https://www.phoronix.com/news/Linux-DRM-Panic-BSoD-Picture

theoparis avatar Jan 20 '25 09:01 theoparis

I have also encountered same issue now.

jmurudi avatar Jan 20 '25 10:01 jmurudi

Working on a fix, I have removed the update from the PPA for now. The fact that selecting an older kernel doesn't work suggests that it might be a device tree problem. I'll let you all know once I know more.

tobhe avatar Jan 20 '25 10:01 tobhe

@theoparis @jmurudi are your devices also MacBook Pro Max M2?

tobhe avatar Jan 20 '25 11:01 tobhe

You could try restoring your previous m1n1 from macos for now and see if that helps.

  1. Find your EFI partition:
diskutil list | grep EFI
   4:                        EFI EFI - UBUNT             500.2 MB   disk0s4
  1. Mount it (don't copy verbatim, make sure the name is matches the partition found in the previous step)
diskutil mount /dev/disk0s4
  1. Backup m1n1
cd /Volumes/EFI\ -\ UBUNT/m1n1
cp boot.bin* ~/
  1. Restore boot.bin.old
mv boot.bin.old boot.bin
  1. Reboot and see if Ubuntu boots now

tobhe avatar Jan 20 '25 11:01 tobhe

You could try restoring your previous m1n1 from macos for now and see if that helps.

  1. Find your EFI partition:
diskutil list | grep EFI
   4:                        EFI EFI - UBUNT             500.2 MB   disk0s4
  1. Mount it (don't copy verbatim, make sure the name is matches the partition found in the previous step)
diskutil mount /dev/disk0s4
  1. Backup m1n1
cd /Volumes/EFI\ -\ UBUNT/m1n1
cp boot.bin* ~/
  1. Restore boot.bin.old
mv boot.bin.old boot.bin
  1. Reboot and see if Ubuntu boots now

It does NOT unfortunately :-( Not even with older kernel. I have even tried upgrading MacOS to latest one.

Image

Rad0van avatar Jan 20 '25 12:01 Rad0van

@Rad0van Looking at the drm panic documentation it looks like adding drm.panic_screen=kmsg to your kernel command line (via grub) might give us some more info.

I have a suspicion what might be causing it. There have been reports of similar errros caused by the controller for the sd-card reader. Try adding module_blacklist=sdhci_pci on the command line, that might fix it.

tobhe avatar Jan 20 '25 13:01 tobhe

@Rad0van Looking at the drm panic documentation it looks like adding drm.panic_screen=kmsg to your kernel command line (via grub) might give us some more info.

I have a suspicion what might be causing it. There have been reports of similar errros caused by the controller for the sd-card reader. Try adding module_blacklist=sdhci_pci on the command line, that might fix it.

Blacklisting sdhci_pci does not help. DRM panic screen works, here's what it shows:

Image

Rad0van avatar Jan 20 '25 14:01 Rad0van

So it isn't what I initially suspected. Do you think you can restore boot.bin from your macos home directory and see if the error is the same (to make sure we didn't break it by switching to the old one)?

tobhe avatar Jan 20 '25 14:01 tobhe

So it isn't what I initially suspected. Do you think you can restore boot.bin from your macos home directory and see if the error is the same (to make sure we didn't break it by switching to the old one)?

AFAICT the same

Image

Rad0van avatar Jan 20 '25 14:01 Rad0van

I'm wondering what could have changed in a way it affects the system even if old(er) m1n1 and kernel is being used. What part of the system is affected / modified as well?

Rad0van avatar Jan 20 '25 15:01 Rad0van

I upgraded from Ubuntu 23.10 today and get exactly the same error on a Mac mini M2

Image

nbosb avatar Jan 20 '25 15:01 nbosb

Thanks for all the help! I think we found the issue and are working on a fix to test. Issue seems to be related to https://github.com/AsahiLinux/linux/commit/90356e15b4f66f7df2fae4cedb791ebbccebde59

@Rad0van I don't really have an explanation for why the old dtb and kernel don't work either but let's hope the fix still works.

tobhe avatar Jan 20 '25 15:01 tobhe

Thanks for all the help! I think we found the issue and are working on a fix to test. Issue seems to be related to AsahiLinux/linux@90356e1

@Rad0van I don't really have an explanation for why the old dtb and kernel don't work either but let's hope the fix still works.

Nice. Would be happy to test. But would be curious on how to actually apply the fix...

Rad0van avatar Jan 20 '25 16:01 Rad0van

Our fix is an updated m1n1binary with the new device-trees included. You can test it by either manually downloading https://files.tobhe.de/ubuntu/m1n1.bin and copying it to your ESP as boot.bin as described above or simply run: curl -F https://files.tobhe.de/ubuntu/fix-m1n1 | sh (from macos) which should do the same with less typing.

tobhe avatar Jan 20 '25 17:01 tobhe

Our fix is an updated m1n1binary with the new device-trees included. You can test it by either manually downloading https://files.tobhe.de/ubuntu/m1n1.bin and copying it to your ESP as boot.bin as described above or simply run: curl -F https://files.tobhe.de/ubuntu/fix-m1n1 | sh (from macos) which should do the same with less typing.

Nice. Is the fix already available? Should I try it?

Rad0van avatar Jan 20 '25 17:01 Rad0van

This should already work, if you can test it that would be very helpful because we can't reproduce it locally (none of us has an m2 max). Once Ubuntu boots again it will probably want to downgrade the kernel. If everything goes well we will push another update including the fix.

tobhe avatar Jan 20 '25 17:01 tobhe

And it WORKS! I needed to apply it manually as my partition is named EFI - ASAHI not EFI - UBUNTU but it works. Thanks a lot!

Rad0van avatar Jan 20 '25 17:01 Rad0van

Great :) I'll try to adjust that script and we will make sure to ship an updated kernel later today. Thank you so much for your help debugging this!

tobhe avatar Jan 20 '25 17:01 tobhe

@Rad0van @nbosb @jmurudi @theoparis - just published a new kernel, 6.12.0-1004.4. It should be available for releases 24.04 and newer. Let me know if you have any issues with it, and thanks for the issue reports!

john-cabaj avatar Jan 21 '25 14:01 john-cabaj

i just updated to the new kernel. it seems to work fine on my machine. thanks for the support and the quick fix!

nbosb avatar Jan 21 '25 14:01 nbosb

Works fine for me.

Rad0van avatar Jan 21 '25 15:01 Rad0van

Hey folks,

I tried spinning up a clean installation of Ubuntu 24.04 Server today on a M1 Max Mac Studio and ran across this exact issue. :( Tries booting, encounters Kernel panic, after a loop of 4-5 reboots with a similar outcome it gets to Recovery Assistant. Image

I already tried applying the fix above but to no success. I can provide anything that'll help debugging if you let me know what & how I should do it.

murarustefaan avatar Feb 08 '25 11:02 murarustefaan

@murarustefaan so as I understand it you had no chance applying the kernel update because it crashes right away? I can respin the server build later to make sure it doesn't include a broken kernel in the first place.

tobhe avatar Feb 08 '25 11:02 tobhe

So it crashes right away, but i tried applying it from MacOS using the curl command a couple of comments above which completed successfully. However, the outcome is the same, the boot process is interrupted by the kernel crash.

murarustefaan avatar Feb 08 '25 11:02 murarustefaan

Ok, I think a freshly built iso with newer kernel could make a difference. I'll let you know once we have something to test. Thanks for the report!

tobhe avatar Feb 08 '25 11:02 tobhe

@murarustefaan new 24.04 server image is up and should include an up-to-date kernel and m1n1.

tobhe avatar Feb 09 '25 12:02 tobhe

Hey @tobhe, thanks for that. Sadly, it's still not working.

I ran a clean installation, deleting all previous partitions and re-installing 24.04-server from scratch. Now instead of the crash screen I initially posted I get the pink screen that was posted above. I also tried re-applying the m1n1 fix above, still to no success.

A couple of reboots into it, it managed to get to this screen, on which it got stuck and had to crtl+alt+delete restart it which led back to the pink panic screen. Image

Let me know if and how I can provide anything else that could help debugging this issue.

murarustefaan avatar Feb 09 '25 13:02 murarustefaan