Kernel panic - not syncing: VFS: unable to mount root fs on unknown-block(8,7)

[slrslr]

Raspbian 10 and RPi4

It happened for several time, that i seen some errors that given me a suggestion like file system disconnected.

But this time after power cycling, it failed to start showing error: Kernel panic - not syncing: VFS: unable to mount root fs on unknown-block(8,7)

as seen on the screenshot:

My system partition is on the external USB drive (made by clonning system SD card using Raspbian built in GUI disk cloner tool and then: sudo sed -i "s/mmcblk0p7/sda7/g" /boot/cmdline.txt It worked good, numerous reboots, but suddenly fails.

What is the easiest way to discover cause or to likely fix it without clean install please?

>>> UPDATE <<<:

what worked was to unmount root partition and run following command on it: e2fsck -f -y -v -C 0 /dev/sda7 (replace partition path if you have different - command: lsblk)

[pelwell]

Is that failure repeatable? You can add framebuffer_height=2160 to get more text on screen at once.

[slrslr]

@pewell unsure what do you mean be repeatable (i tried to power cacle RPi several times with drive connected to various USB ports and error seems same. Regarding framebuffer_height, i added that line to config.txt in the boot partition, but i do not see any change in what is displayed.

[pelwell]

When I say repeatable I mean that it isn't a one-off error. Setting framebuffer_height=2160 in config.txt increases the number of lines on screen for me on a Pi 4, but only in the early stages of booting.

[slrslr]

I have limitted time on this issue (then i will erase the drive), so i would appreaciate if someone tell me exact commands and steps. For stupid people i say i am just GUI user, which means obvious things are not obvious.

[slrslr]

For all including @pelwell i edited wrong file on different drive, i edited correct one now and here is what i see with increased framebuffer_height. I hope i have not wasted time and someone will try to fix the problem. Or suggest how to try to fix it. Thank you

[pelwell]

That's much more useful. The other thing you could try is adding maxcpus=1 to cmdline.txt (being careful to keep it all in one line), which will remove the stack traces for the other cores.

[slrslr]

@pelwell this is how it looks like with "maxcpus=1". When i redefine root= path to the partition on the SD card (mmcblk0p7) instead of sda7 then system boot up, but mouse cursor not moves and keyboard does not react. But that is maybe different issue, i want it boot from sda7 to be fixed.

[pelwell]

The list is clearly showing sda7 as an option, but you are saying that root=sda7 doesn't work? Is it a timing issue? What do you have in cmdline.txt?

[slrslr]

Is it a timing issue?

Please be more descriptive on how exactly i find out that. I already said i am an GUI user.

[pelwell]

My comment was more thinking aloud than a question. What is in your cmdline.txt?

[slrslr]

What is in your cmdline.txt?

console=serial0,115200 console=tty3 root=/dev/mmcblk0p7 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

it was root=/dev/sda7 but shows errors mentioned. when it is mmcblk0p7 then mouse and keyb. does not react.

[SOI2016]

step :1) vim /boot/cofig.txt

add below 2 lines

enable_uart=1 dtoverlay=pi3-miniuart-bt

step :2) vim /boot/cmdline.txt

delete all lines in the file and add below line . mmcblk0p2 is my partition . I think so, you need to edit the partition.

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

step : 3) vim /etc/inittab #add below 1 line T0:23:respawn:/sbin/getty -L ttyAMA0 115200 vt100

[ED6E0F17]

It seems to be an unfortunate fact that the kernel gives the same error for both Device Not Present, and File System Unreadable: https://wiki.gentoo.org/wiki/Knowledge_Base:Unable_to_mount_root_fs

"rootwait" in the cmdline should wait for the device to appear, and "block(8,7)" IS "/dev/sda7" so the implication is that the filesystem is not being recognised for some reason.

Can you boot from SDcard and than see what "dmesg" says about the usb drive?

[slrslr]

@ED6E0F17 Thank you. In an attempt to discover dmesg output i booted using SD card's root partition (in /boot/cmdline.txt) instead of external USB drive /dev/sda7 root partition (which fails as per this issue). The keyboard and mouse does not react, but the SSH worked and here is dmesg output (p@@sw0rd is: rp) as you wanted.

I then tried to set root partition to external USB drive and remove rootwait boot parameter: sudo sed -i "s/mmcblk0p7/sda7/g" /boot/cmdline.txt sudo sed -i "s/ rootwait//g" /boot/cmdline.txt it ended up with fs error - unable to mount root fs on unknown-block(0,0)

@SOI2016 /boot/cofig.txt already contained dtoverlay line so i commented it and pasted your lines. /boot/cmdline.txt i backed up original file and then emptied it, pasted your lines and replaced by my sd card root partition, mouse, keyb. not reacted as before.. then i used USB HDD root partition, and it failed with unable to mount fs unknown-block(8,7) as shown on the screenshot from the first post.

[ED6E0F17]

Mouse and keyboard are detected, so it is odd that you cant use those.

The ssd adaptor is rated at 900 mA, so you will need to use an external power supply, even if it says that you don`t.

I half expected to see the drive get automatically mounted. I would have suggested that you use "gparted" to test/repair it, but without a mouse you are left with some unpleasant command line alternatives.

[slrslr]

@ED6E0F17 thanks for the suggestions. Following worked to make sda7 partition bootable again. Regarding gparted, i think it is using fsck, so i wanted to try to fix it with basic tools that are already part of the Raspbian 10, so i did this: in /boot/cmdline.txt set root to be SD card, not my external drive sda. The grub boot line was missing "fsck.mode=force fsck.repair=yes" parameters btw, maybe it means that auto mode is used, but even when i forced it using that parameters later, it was unable to repair the invalid superblock issue of the /dev/sda7.

So RPi booted from SD and via SSH i ran: lsblk i see external drive is attached, so i tried to unmount root partition on it and scan it: umount /dev/sda sudo fsck -v /dev/sda

fsck from util-linux 2.33.1 e2fsck 1.44.5 (15-Dec-2018) ext2fs_open2: Bad magic number in super-block fsck.ext2: Superblock invalid, trying backup blocks... fsck.ext2: Bad magic number in super-block while trying to open /dev/sda

The superblock could not be read or does not describe a valid ext2/ext3/ext4 filesystem. If the device is valid and it really contains an ext2/ext3/ext4 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock: e2fsck -b 8193 or e2fsck -b 32768

Found a dos partition table in /dev/sda

dmesg command outputs:

[ 69.261724] EXT4-fs (sda7): VFS: Found ext4 filesystem with invalid superblock checksum. Run e2fsck? [ 69.280439] FAT-fs (sda6): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

tried: sudo e2fsck -b 8193 /dev/sda sudo e2fsck -b 32768 /dev/sda but it continue to output same error as fsck command above

sudo mke2fs -n /dev/sda

mke2fs 1.44.5 (15-Dec-2018) Found a dos partition table in /dev/sda Creating filesystem with 29305205 4k blocks and 7331840 inodes Filesystem UUID: e76f087f-848d-4f48-b0cc-3c4c7a1b17f6 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872

Then i risked and ran command: sudo fsck -v -b 32768 /dev/sda7

i hold enter for maybe 30 seconds confirming tons of prompts of fixing "Free inodes count wrong for group xy".

result:

root: ***** FILE SYSTEM WAS MODIFIED *****

  155391 inodes used (2.17%, out of 7151616)
     837 non-contiguous files (0.5%)
      67 non-contiguous directories (0.0%)
         # of inodes with ind/dind/tind blocks: 0/0/0
         Extent depth histogram: 142377/34
25735444 blocks used (89.97%, out of 28605813)
       0 bad blocks
       2 large files

  128513 regular files
   13704 directories
       7 character device files
       0 block device files
       0 fifos
    1155 links
   13149 symbolic links (12956 fast symbolic links)
       9 sockets

---

  156537 files

when mounting /dev/sda, it shows wrong fs type error (likely i had to use not sda but sda7)

then i installed gparted (apt install gparted) and ran it in GUI mode, selected sda drive and clicked sda7 partition and selected to check it, it ran: e2fsck -f -y -v -C 0 /dev/sda7 (later i found that this e2fsck command alone is sufficing to fix the issue described on this page). I am able to mount root partition of the external drive: sudo mount /dev/sda7 /mnt/

and see contents and when i added /dev/sda7 into root boot parameter of the /boot/cmdline.txt it now boots OK.

I would welcome feedback. To close this issue, i am wondering if Raspbian can be made to automatically check FS at boot using e2fsck -f -y -v -C 0 /dev/sda7 to repair invalid super block?

[slrslr]

In my previous reply in the last paragraph is what worked and a question for the developers. @raspberrypi

[pelwell]

sda is the whole device, starting with the partition table - that was never going to work as a file system. The kernel is configured to run an fsck at boot if the volume wasn't cleanly unmounted, and after some number of boots. Google suggests you can request an fsck on boot by adding fsck.mode=force to cmdline.txt. You can optionally force errors to be fixed with fsck.repair=yes.

[slrslr]

@pewell thx for the feedback. yes i am aware about these boot parameters, though as mentioned earlier in this thread, "it was unable to repair the invalid superblock issue of the /dev/sda7".

[pelwell]

If fsck can't fix it then I don't think there's anything we can do automatically.

[pelwell]

By the way, adding critical information by editing comments makes it invisible to anyone reading the email thread, as I was.

[Neddily]

Thanks so much for this, slrslr. A power outage last night caused the same issue for me, such that my Pi 4 would no longer boot from its 512GB SSD. After finding this thread I booted from my backup SD card, inserted the SSD and verified it was at least visible as on the USB bus using lslbk. Then,

$ sudo e2fsck -f -y -v -C 0 /dev/sdb2

successfully ran and fixed a bunch of inodes. I shut down, removed the SD card then successfully rebooted from the SSD. All files intact. Phew.

It's all pretty much black magic to me at this point, but now I'm motivated to a) figure out what this command did, and b) find a decent UPS for my Pi.

[ghost]

It's all pretty much black magic to me at this point, but now I'm motivated to a) figure out what this command did, ...

May I suggest man e2fsck as a good place to start 😉

[Neddily]

Thanks yes, I did do that but still don't grok it - it's a maze of twisty passages all alike. I also wonder what happened to my Pi's SSD during the overnite lightning storm, how, exactly, it got corrupted, and how I might prevent it in the future. My end game is to store my own media files cloudlessly at home.

[mihaimiculescu]

Thanks @slrslr for getting at the bottom of this. After brownout (very short blackout) ran into it big time. Sorted @slrslr rules!

[mihaimiculescu]

There is another Kernel Panic situation deriving from boot partition getting corrupted. Should this be the subject matter of another issue?

[pelwell]

Yes, different problems need different Issues.

[pelwell]

Mon français n'est pas formidable, donc en anglais...

Can you install any Raspberry Pi image to an SD card (32-bit RPiOS Lite is enough), allow it to boot, then attach your SSD. You should then be able to use fsck.

[flowaves]

Merci, et quelle est la commande exacte à exécuter ?

[pelwell]

1. Remove any hardware except the monitor and keyboard.
2. Insert a blank SD card.
3. Plug in the power.
4. Use the Shift key to start the network installer.
5. Follow the instructions (you can switch to French).