operating-system icon indicating copy to clipboard operation
operating-system copied to clipboard

Raspberry Pi3B+ + USB SSD Drive - Update to HassOS 5.8 fail

Open noide35 opened this issue 4 years ago • 79 comments

Hello,

just to Notify that I tried this night to upgrade my HA (Hass OS 4.17) installed on my PI3B+ to last Hass OS 5.8. I'm using my PI3B+ with a SSD drive connected on USB. This installation totally broke my setup : my raspberry became totally unresponsive 20 minutes waiting after the beginning of the upgrade. After that, I tried a fresh install of Hass OS 5.8 but also without success. Unfortunately I don't have any logs as I lost access to my PI and I have no screen connected to it.

I was able to get back a HA setup only by doing a fresh install of Hass OS 4.16 (the only one I had) and installing a snapshot on it.

So for now and from what I see, it seems there is some issues with this new Hass OS version and PI with USB boot setup.

Again sorry for not having any logs...

noide35 avatar Dec 16 '20 00:12 noide35

In my odroid n2 I have a similar problem. It is also stuck most of the times. Sometimes it finish booting but than minutes later the memory consumption spikes and the system is unresponsive again. Could be similar problem?

I'll try to get some logs on the next successful boot

liudger avatar Dec 16 '20 07:12 liudger

@noide35 you can download older releases from the GitHub releases page: https://github.com/home-assistant/operating-system/releases

agners avatar Dec 16 '20 10:12 agners

@agners Thanks, I missed that. It was late and I needed to quickly put a version running, so that was fine for me to go back to 4.16.

So for now I have Hass OS 4.16 with HA Core 2020.12.0 running well.

I also saw others issues with version 5.x of HA with PI and USB stick/drive connected. I think I'll have time during next weekend to do more tests and may be gather logs.

What I think I'll do :

  • Try again a fresh install of Hass OS 5.8 and gathering logs with USB/SSD drive
  • Try a fresh install of Hass OS 5.8 on SD card
  • I'll also try a 4.20 version on SSD/USB drive

noide35 avatar Dec 16 '20 10:12 noide35

had the exact same problem: upgrade failed - complete re-flash of new downloaded 5.8 version didn't work (hassos_rpi4-64-5.8.img.xz) - powering my pi 3b+ not even the green light flashed. after using a new flashed version of hassos_rpi3-64-4.16.img did the job. now my pi starts again... will try and update from a fresh installation... let's see if it works...

sir106 avatar Dec 16 '20 11:12 sir106

My Pi (4) does not even try to startup after upgrading 5.5 to 5.8. As I have a full USB-SSD installation, how can I install 5.5 or 5.7 without having to flash the whole SSD again? I probably have a backup but I would prefer having a rescue image without wiping the whole drive...

I wont update ever again after this 😅

Is there a way to use dd to clone a specific partition from a SD card to the SSD? 🤔

Edit: well, it seems that the update killed the whole partition table or boot partition as the pi does not even try to boot from USB SSD. Sadly I dont have an HDMI adapter and thus I am not able to see what the problem is. Maybe there is a way to just fix the boot partition with dd instead of flashing the whole system?

boesing avatar Dec 16 '20 23:12 boesing

@boesing I don't think its related the partition label. More likely you are hitting issue #1045: We deploy a new firmware for Raspberry Pi 4, and it seems to break USB SSD boot for some USB SSD enclosures.

The best way to recover is to plug-in the USB SSD to your computer, and browse the first partition (FAT partition). It should contain files like config.txt delete pieeprom.sig and pieeprom.upd from it. Then use Raspberry Pi OS to downgrade to an older Raspberry Pi firmware (e.g. 2020-09-03).

This issue is about Raspberry Pi 3.

agners avatar Dec 17 '20 12:12 agners

Same problem. I use RPi 3 with USB SSD and it stopped working after lastest OS update. Connected to the TV I saw a black screen and "no signal" message on TV

fabiosci avatar Dec 17 '20 13:12 fabiosci

had the exact same problem: upgrade failed - complete re-flash of new downloaded 5.8 version didn't work (hassos_rpi4-64-5.8.img.xz)

@sir106 you flashed the image for Raspberry Pi 4 and expected it to work on Raspberry Pi 3+?

agners avatar Dec 17 '20 14:12 agners

@fabiosci which version have you been using? 32-bit or 64-bit?

agners avatar Dec 17 '20 14:12 agners

Upgrade from 4.17 to 5.8 works and booted on my end using a Samsung 64GB USB flash drive on a Raspberry Pi 3+, so it seems this depends on the environment :-(

There could be several things which might be the culprit: Firmware, U-Boot or the new Linux Kernel 5.4. What would be interesting is to test different combinations of firmware/U-Boot with the 5.8 release: E.g. to test the old firmware, copy start.elf, fixup.dat and bootcode.bin from the first boot partition to the same place of a 5.8 USB drive. To test U-Boot from 4.17 copy u-boot.bin and boot.scr to the same place of a 5.8 USB drive.

When the screen turns black, it must be either the Raspberry Pi firmware or U-Boot which is causing issues. What would be interesting is to use the firmware

agners avatar Dec 17 '20 15:12 agners

Hello Stefan,

Thanks for this first analysis. I'll test ASAP what you're suggesting and will get back to you with informations.

To give you a little bit more info on my side, I'm using 32 bit image and when the issue appears, "ACT" LED of the Pi stays off (not blincking at all).

On my side I'm using these two parts : https://www.amazon.fr/gp/product/B011O9JAXS https://www.amazon.fr/gp/product/B01KFLH1WS

It's dirrectly connected to the Pi without external USB hub.

noide35 avatar Dec 17 '20 15:12 noide35

@fabiosci which version have you been using? 32-bit or 64-bit?

32bit

fabiosci avatar Dec 17 '20 15:12 fabiosci

I have problems with 5.8 as well. Upgrade from the web inteface ended with a heavily corrupted partition 8. Install to a fresh 32GB microSDHC card ended by a corrupted partition 8 as well, and additionally Card never left busy state, error -110. Returning back the original 16GB card, again corrupted partition 8. Instaliing fresh hassos_rpi3-4.20.img.gz works.

stanislav-brabec avatar Dec 17 '20 20:12 stanislav-brabec

had the exact same problem: upgrade failed - complete re-flash of new downloaded 5.8 version didn't work (hassos_rpi4-64-5.8.img.xz)

@sir106 you flashed the image for Raspberry Pi 4 and expected it to work on Raspberry Pi 3+?

thanks for pointing it out!my fault... will try it again - was too quick when selecting the download ;)

sir106 avatar Dec 17 '20 20:12 sir106

Have you tried the hdd in usb2 slot?

I've been having a very similar action and although the drives are recognized within raspberry pi (checked a few different USB caddys due to the issues some have), it'll boot when the SSD is in the USB2 slot and not in the USB3 slot.

May be a driver issue.

Have also seen issues with the power supply on the new Kernel. Before updating POE splitter was working fine, now it wants me to use my standard Pi Power Supply, otherwise there are issues.

wlatic avatar Dec 18 '20 05:12 wlatic

@wlatic I don't get your point there, Raspberry 3B+ has only USB2 ports and there no PoE at all in the loop as I'm using standard power supply (3A)

noide35 avatar Dec 18 '20 07:12 noide35

I suspect that the source of all problems is a massive data corruption somewhere in the kernel. While trying to recover the crashed system, I have found that the first kilobyte of part8:supervisor/homeassistant/.storage/lovelace file is corrupted by a ~20 random single byte changes. Also the end of the part8:supervisor/homeassistant/home-assistant_v2.db is corrupted in a similar way.

The most often corruption bytes 0x00, 0x60, 0x7B, 0x22, 0x24, and less frequently other bytes.

stanislav-brabec avatar Dec 20 '20 02:12 stanislav-brabec

Hello everybody,

Hope you're doing well.

@agners You were right ! I've done som tests this morning and here are the results : 4.20 firmware (I never tested) => OK, working as expected 5.8 native (to test again and to be sure problem is still there) => KO ; one blink of ACT LED at power up, black screen and no more activity of ACT LED. 5.8 with firmware of 4.20 (start.elf, fixup.dat and bootcode.bin) => boot ok !

So it's seems to be an issue with firmware included in HasssOS 5.8

noide35 avatar Dec 20 '20 11:12 noide35

Hello again,

I've just done one more test as I saw version 5.9 was available :

5.9 native (to be sure problem is still there) => KO ; one blink of ACT LED at power up, black screen and no more activity of ACT LED. 5.9 with firmware of 4.20 (start.elf, fixup.dat and bootcode.bin) => boot ok !

So same issue with HassOS 5.9.

noide35 avatar Dec 20 '20 11:12 noide35

My raspberry 4 with Kingston mSATA 120 GB SSD x586 card from GeekWorm works perfect with 5.8. My raspberry 3B+ with Kingston mSATA 120 GB on GeekWorm x850 v3.1 doesn’t work above 5.3 neither 32 or 64 bit, however it works fine with a Sandisk USB stick. I did a lot of test and made 5.8 work (and 5.4…) I just copied the “boot” files on the FAT partition from 5.3 to the 5.8 version and now it works fine. I believe the problem depends on WHICH USB SSD you have and obviously something has changed with the boot files from 5.4 on onwards. I hope this helps anyone. Gunnar

GuruLarsson avatar Dec 21 '20 07:12 GuruLarsson

My raspberry 4 with Kingston mSATA 120 GB SSD x586 card from GeekWorm works perfect with 5.8. My raspberry 3B+ with Kingston mSATA 120 GB on GeekWorm x850 v3.1 doesn’t work above 5.3 neither 32 or 64 bit, however it works fine with a Sandisk USB stick. I did a lot of test and made 5.8 work (and 5.4…) I just copied the “boot” files on the FAT partition from 5.3 to the 5.8 version and now it works fine. I believe the problem depends on WHICH USB SSD you have and obviously something has changed with the boot files from 5.4 on onwards. I hope this helps anyone. Gunnar

Thanks Gunnar for the info.. i was about to loose all hope :)

After banging my head against the wall trying to get RPI3B (not plus version) to boot HA from USB SSD i confirm this works for 5.9 (and most likely all versions 5.4+ onwards) . Just copy the boot files from the 5.3 FAT partition over 5.9. In case it is of any interest i'm using USB SSD adapter which is based on JMICRON JMS580 and SSD is Samsung 860 EVO SSD 250 Gt M.2 SATA SSD

pxFIN avatar Dec 23 '20 16:12 pxFIN

My raspberry 4 with Kingston mSATA 120 GB SSD x586 card from GeekWorm works perfect with 5.8. My raspberry 3B+ with Kingston mSATA 120 GB on GeekWorm x850 v3.1 doesn’t work above 5.3 neither 32 or 64 bit, however it works fine with a Sandisk USB stick. I did a lot of test and made 5.8 work (and 5.4…) I just copied the “boot” files on the FAT partition from 5.3 to the 5.8 version and now it works fine. I believe the problem depends on WHICH USB SSD you have and obviously something has changed with the boot files from 5.4 on onwards. I hope this helps anyone. Gunnar

Thanks Gunnar for the info.. i was about to loose all hope :)

After banging my head against the wall trying to get RPI3B (not plus version) to boot HA from USB SSD i confirm this works for 5.9 (and most likely all versions 5.4+ onwards) . Just copy the boot files from the 5.3 FAT partition over 5.9. In case it is of any interest i'm using USB SSD adapter which is based on JMICRON JMS580 and SSD is Samsung 860 EVO SSD 250 Gt M.2 SATA SSD

How did you do this ? "Just copy the boot files from the 5.3 FAT partition over 5.9" Because with hassio, we can't see the content of my ssd on my Mac.

floow77 avatar Dec 29 '20 16:12 floow77

Can confirm I have the same issue not able to upgrade from the HA UI. In my case from 4.13 to 5.9 (no luck with a fresh 5.9 either) I'm running a RPi3B+ with a WD Green 120GB SSD and a StarTech USB312SAT3CB sata2usb adapter

eketjall avatar Dec 31 '20 11:12 eketjall

Same issue here with a fresh 5.9 image running on a RPi3B with Kingston A400 and a StarTech USB312SAT3CB sata2usb adapter. Fails to boot.

Replaced start.elf, fixup.dat and bootcode.bin from the 4.20 image and now all is well with 5.9. Thanks for testing that @noide35

rgreuel avatar Dec 31 '20 22:12 rgreuel

My raspberry 4 with Kingston mSATA 120 GB SSD x586 card from GeekWorm works perfect with 5.8. My raspberry 3B+ with Kingston mSATA 120 GB on GeekWorm x850 v3.1 doesn’t work above 5.3 neither 32 or 64 bit, however it works fine with a Sandisk USB stick. I did a lot of test and made 5.8 work (and 5.4…) I just copied the “boot” files on the FAT partition from 5.3 to the 5.8 version and now it works fine. I believe the problem depends on WHICH USB SSD you have and obviously something has changed with the boot files from 5.4 on onwards. I hope this helps anyone. Gunnar

Thanks Gunnar for the info.. i was about to loose all hope :) After banging my head against the wall trying to get RPI3B (not plus version) to boot HA from USB SSD i confirm this works for 5.9 (and most likely all versions 5.4+ onwards) . Just copy the boot files from the 5.3 FAT partition over 5.9. In case it is of any interest i'm using USB SSD adapter which is based on JMICRON JMS580 and SSD is Samsung 860 EVO SSD 250 Gt M.2 SATA SSD

How did you do this ? "Just copy the boot files from the 5.3 FAT partition over 5.9" Because with hassio, we can't see the content of my ssd on my Mac.

I'm using HassOS installation.. so when you plug your USB SSD in Windows.. the boot partition 'mounts' and you can see all the RPI boot files.. basically you can also mount HassOS partitions different way.. run for example linux and use mount from command line

pxFIN avatar Jan 01 '21 08:01 pxFIN

Same for me here with a SanDisk Plus 120GB and a Renkforce 2,5 Case

Even with OS 5.10 no chance to boot without hacking, so i changed back to 4.20 stable https://github.com/home-assistant/operating-system/releases/tag/4.20

upwindanderl avatar Jan 02 '21 15:01 upwindanderl

Here is a simple, reliable workaround using an shim bootloader on a SD card (referenced as Special bootcode.bin-only boot mode:

Format a SD card as FAT 32 and put the latest raspberry bootcode.bin on it.

The Pi will then boot from the sd card which will immediately hand over to the bootloader on the ssd. This is a brief read-only access so the normal reliability issues shouldn't apply. Since the sd card is never mounted and thus invisible to homeassistant this will survive upgrades.

Update: Other users have reported that this method does not work for them. It works for me, but is apparently not as reliable as i thought.

gregod avatar Jan 08 '21 14:01 gregod

Hello everyone,

I have been suffering from the same issue reported here, but it took me a while to realise it was an issue with Home Assistant because I was just trying to move from SD cards to SSD over USB on a new USB adaptor. At first I thought it was a problem with the adaptor itself, but then I found out that Raspbbian (Raspios) was booting properly from the same disk / connector.

I am running on RPI 3B (not plus) and I did the required OTP changes to enable USB boot. The interesting thing was that either 5.9 or 5.10 didn't manage to boot most of the time, but sometimes and quite randomly the Pi was booting up successfully. I can confirmed that since switching to 5.3 the USB boot works reliably every time, without the need to have a special bootloader on an SD card.

During the troubleshooting I hooked up a UART serial cable to the RPI and was booting with a modified bootloader.bin from an SD card in order to enable the UART console during the boot sequence, and foun the following.

This is the log from a successful boot:

Raspberry Pi Bootcode

Read File: config.txt, 1827

Read File: start.elf, 2949600 (bytes)

Read File: fixup.dat, 7289 (bytes)

MESS:00:00:04.891066:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:04.895956:0: brfs: File read: 1827 bytes
MESS:00:00:04.948950:0: HDMI0:EDID error reading EDID block 0 attempt 0
MESS:00:00:04.955112:0: HDMI0:EDID error reading EDID block 0 attempt 1
MESS:00:00:04.961449:0: HDMI0:EDID error reading EDID block 0 attempt 2
MESS:00:00:04.967786:0: HDMI0:EDID error reading EDID block 0 attempt 3
MESS:00:00:04.974123:0: HDMI0:EDID error reading EDID block 0 attempt 4
MESS:00:00:04.980459:0: HDMI0:EDID error reading EDID block 0 attempt 5
MESS:00:00:04.986797:0: HDMI0:EDID error reading EDID block 0 attempt 6
MESS:00:00:04.993133:0: HDMI0:EDID error reading EDID block 0 attempt 7
MESS:00:00:04.999470:0: HDMI0:EDID error reading EDID block 0 attempt 8
MESS:00:00:05.005807:0: HDMI0:EDID error reading EDID block 0 attempt 9
MESS:00:00:05.011902:0: HDMI0:EDID giving up on reading EDID block 0
MESS:00:00:05.043204:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:05.048023:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:05.072746:0: brfs: File read: 1827 bytes
MESS:00:00:05.266608:0: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
MESS:00:00:05.274060:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:05.280084:0: *** Restart logging
MESS:00:00:05.284683:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 0
MESS:00:00:05.292052:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 1
MESS:00:00:05.298910:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 2
MESS:00:00:05.305769:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 3
MESS:00:00:05.312625:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 4
MESS:00:00:05.319483:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 5
MESS:00:00:05.326342:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 6
MESS:00:00:05.333198:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 7
MESS:00:00:05.340056:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 8
MESS:00:00:05.346915:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 9
MESS:00:00:05.353529:0: hdmi: HDMI0:EDID giving up on reading EDID block 0
MESS:00:00:05.359432:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 0
MESS:00:00:05.367226:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 1
MESS:00:00:05.374084:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 2
MESS:00:00:05.380942:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 3
MESS:00:00:05.387800:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 4
MESS:00:00:05.394657:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 5
MESS:00:00:05.401515:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 6
MESS:00:00:05.408372:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 7
MESS:00:00:05.415230:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 8
MESS:00:00:05.422087:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 9
MESS:00:00:05.428703:0: hdmi: HDMI0:EDID giving up on reading EDID block 0
MESS:00:00:05.434576:0: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
MESS:00:00:05.443064:0: HDMI0: hdmi_pixel_encoding: 162000000
MESS:00:00:05.448756:0: vec: vec_middleware_power_on: vec_base: 0x7e806000 rev-id 0x00002708 @ vec: 0x7e806100 @ 0x00000420 enc: 0x7e806060 @ 0x00000220 cgmsae: 0x7e80605c @ 0x00000000
MESS:00:00:05.470073:0: dtb_file 'bcm2710-rpi-3-b.dtb'
MESS:00:00:05.513395:0: brfs: File read: /mfs/sd/bcm2710-rpi-3-b.dtb
MESS:00:00:05.518058:0: Loading 'bcm2710-rpi-3-b.dtb' to 0x100 size 0x6d40
MESS:00:00:05.538014:0: brfs: File read: 27968 bytes
MESS:00:00:06.088722:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:06.093246:0: dtparam: audio=on
MESS:00:00:06.112323:0: brfs: File read: 1827 bytes
MESS:00:00:06.128466:0: brfs: File read: /mfs/sd/cmdline.txt
MESS:00:00:06.132430:0: Read command line from file 'cmdline.txt':
MESS:00:00:06.138317:0: 'dwc_otg.lpm_enable=0 console=tty1'
MESS:00:00:06.159418:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
MESS:00:00:07.628019:0: brfs: File read: 34 bytes
MESS:00:00:07.773572:0: brfs: File read: /mfs/sd/u-boot.bin
MESS:00:00:07.777435:0: Loading 'u-boot.bin' to 0x8000 size 0x61f0c
MESS:00:00:07.783434:0: Device tree loaded to 0x2eff8e00 (size 0x7118)
MESS:00:00:07.790695:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:07.797357:0: uart: Baud rate change done...
MESS:00:00:07.800771:0: uart: Baud rate

U-Boot 2020.10 (Jan 01 2021 - 15:39:17 +0000)

DRAM:  948 MiB
RPI 3 Model B (0xa02082)
MMC:   mmc@7e202000: 0, mmcnr@7e300000: 1
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
starting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 4 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
switch to partitions #0, OK
mmc0 is current device
** No partition table - mmc 0 **
Card did not respond to voltage select!

Device 0: Vendor: ASMT     Rev: 0    Prod: 2115            
            Type: Hard Disk
            Capacity: 228936.5 MB = 223.5 GB (468862128 x 512)
... is now current device
Scanning usb 0:1...
Found U-Boot script /boot.scr
2421 bytes read in 1 ms (2.3 MiB/s)
## Executing script at 02400000

Device 0: Vendor: ASMT     Rev: 0    Prod: 2115            
            Type: Hard Disk
            Capacity: 228936.5 MB = 223.5 GB (468862128 x 512)
... is now current device
loading env...

usb read: device 0 block # 1214464, count 32 ... 32 blocks read: OK
## Error: bad CRC, import failed
Trying to boot slot A, 2 attempts remaining. Loading kernel ...
7639672 bytes read in 191 ms (38.1 MiB/s)
storing env...

usb write: device 0 block # 1214464, count 32 ... 32 blocks written: OK
Starting kernel
Kernel image @ 0x080000 [ 0x000000 - 0x749278 ]
## Flattened Device Tree blob at 2eff8e00
   Booting using the fdt blob at 0x2eff8e00
   Using Device Tree in place at 2eff8e00, end 2f002f17

Starting kernel ...

This is the log from an unsuccessful boot:

Raspberry Pi Bootcode

Read File: config.txt, 1827

Read File: start.elf, 2949600 (bytes)

Read File: fixup.dat, 7289 (bytes)

MESS:00:00:05.811133:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:05.816025:0: brfs: File read: 1827 bytes
MESS:00:00:05.921275:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:05.926099:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:05.950823:0: brfs: File read: 1827 bytes
MESS:00:00:06.146568:0: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
MESS:00:00:06.154020:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:06.160044:0: *** Restart logging
MESS:00:00:06.196166:0: HDMI0: hdmi_pixel_encoding: 162000000
MESS:00:00:06.205444:0: dtb_file 'bcm2710-rpi-3-b.dtb'
MESS:00:00:06.248811:0: brfs: File read: /mfs/sd/bcm2710-rpi-3-b.dtb
MESS:00:00:06.253471:0: Loading 'bcm2710-rpi-3-b.dtb' to 0x100 size 0x6d40
MESS:00:00:06.273171:0: brfs: File read: 27968 bytes
MESS:00:00:06.825014:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:06.829550:0: dtparam: audio=on
MESS:00:00:06.848617:0: brfs: File read: 1827 bytes
MESS:00:00:06.864757:0: brfs: File read: /mfs/sd/cmdline.txt
MESS:00:00:06.868726:0: Read command line from file 'cmdline.txt':
MESS:00:00:06.874615:0: 'dwc_otg.lpm_enable=0 console=tty1'
MESS:00:00:06.895747:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
MESS:00:00:08.372110:0: brfs: File read: 34 bytes
MESS:00:00:08.517615:0: brfs: File read: /mfs/sd/u-boot.bin
MESS:00:00:08.521476:0: Loading 'u-boot.bin' to 0x8000 size 0x61f0c
MESS:00:00:08.527475:0: Device tree loaded to 0x2eff8e00 (size 0x711c)
MESS:00:00:08.534745:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:08.541402:0: uart: Baud rate change done...
MESS:00:00:08.544813:0: uart: Baud rate

U-Boot 2020.10 (Jan 01 2021 - 15:39:17 +0000)

DRAM:  948 MiB
RPI 3 Model B (0xa02082)
MMC:   mmc@7e202000: 0, mmcnr@7e300000: 1
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
starting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
switch to partitions #0, OK
mmc0 is current device
** No partition table - mmc 0 **
Card did not respond to voltage select!

Device 0: unknown device
Waiting for Ethernet connection... unable to connect.
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/01-b8-27-eb-67-f6-d4
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/00000000
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/0000000
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/000000
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/00000
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/0000
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/000
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/00
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/0
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/default-arm-bcm283x-rpi
Waiting for Ethernet connection... unable to connect.
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/default-arm-bcm283x
Waiting for Ethernet connection...  

U-Boot 2020.10 (Jan 01 2021 - 15:39:17 +0000)

DRAM:  948 MiB
RPI 3 Model B (0xa02082)
MMC:   mmc@7e202000: 0, mmcnr@7e300000: 1
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
starting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
WARNING at drivers/mmc/bcm2835_sdhost.c:414/bcm2835_send_command()!
WARNING at drivers/mmc/bcm2835_sdhost.c:414/bcm2835_send_command()!
Card did not respond to voltage select!
Card did not respond to voltage select!

Device 0: unknown device
Waiting for Ethernet connection... unable to connect.
missing environment variable: pxeuuid
missing environment variable: bootfile
Retrieving file: pxelinux.cfg/01-b8-27-eb-67-f6-d4

Basically it looks like U-Boot only saw 3 USB devices on the failed attempt, while the successful attempt reported 4 USB devices, including the SSD drive. After the PXE boot attempt time out I got an HassOS> prompt where I tried to issue an USB reset multiple times, all of them failing to detect the attached drive:

HassOS> usb reset
resetting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 3 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found

The following is the output from usb tree

HassOS> usb tree
USB device tree:
  1  Hub (480 Mb/s, 0mA)
  |   U-Boot Root Hub 
  |
  |+-2  Hub (480 Mb/s, 2mA)
    |
    |+-3  Vendor specific (480 Mb/s, 2mA)

But then I tried a usb stop followed by a usb start and here is what happened:

HassOS> usb stop
stopping USB..
HassOS> use   b start
starting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 4 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
HassOS> usb tree
USB device tree:
  1  Hub (480 Mb/s, 0mA)
  |   U-Boot Root Hub 
  |
  |+-2  Hub (480 Mb/s, 2mA)
    |
    |+-3  Vendor specific (480 Mb/s, 2mA)
    |  
    |+-4  Mass Storage (480 Mb/s, 0mA)
         ASMedia AS2115 00000000000000000000

Following is the output from usb info after reinitializing the USB stack

1: Hub,  USB Revision 1.10
 -  U-Boot Root Hub 
 - Class: Hub
 - PacketSize: 8  Configurations: 1
 - Vendor: 0x0000  Product 0x0000 Version 0.0
   Configuration: 1
   - Interfaces: 1 Self Powered 0mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 1
     - Class Hub
     - Endpoint 1 In Interrupt MaxPacket 2 Interval 255ms

2: Hub,  USB Revision 2.0
 - Class: Hub
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x0424  Product 0x9514 Version 2.0
   Configuration: 1
   - Interfaces: 1 Self Powered Remote Wakeup 2mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 1
     - Class Hub
     - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms
     - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms

3: Vendor specific,  USB Revision 2.0
 - Class: Vendor specific
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x0424  Product 0xec00 Version 2.0
   Configuration: 1
   - Interfaces: 1 Self Powered Remote Wakeup 2mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 3
     - Class Vendor specific
     - Endpoint 1 In Bulk MaxPacket 512
     - Endpoint 2 Out Bulk MaxPacket 512
     - Endpoint 3 In Interrupt MaxPacket 16 Interval 4ms

4: Mass Storage,  USB Revision 2.10
 - ASMedia AS2115 00000000000000000000
 - Class: (from Interface) Mass Storage
 - PacketSize: 64  Configurations: 1
 - Vendor: 0x174c  Product 0x1153 Version 0.1
   Configuration: 1
   - Interfaces: 1 Self Powered 0mA
     Interface: 0
     - Alternate Setting 0, Endpoints: 2
     - Class Mass Storage, Transp. SCSI, Bulk only
     - Endpoint 1 In Bulk MaxPacket 512
     - Endpoint 2 Out Bulk MaxPacket 512

I then tried to run usbboot without success because I could not figure out a way to select the device to boot from, and could not really find any online documentation.

Given all of the above I would tend to think that the issue is with U-Boot, and I hope that the debugging data might be useful to track down the issue. Now the device is happily running on 5.3 and attached to my production instance of Home Assistant, but let me know if I can provide you more data to help troubleshooting

piffio avatar Jan 09 '21 21:01 piffio

Here is a simple, reliable workaround using an shim bootloader on a SD card (referenced as Special bootcode.bin-only boot mode:

Format a SD card as FAT 32 and put the latest raspberry bootcode.bin on it.

The Pi will then boot from the sd card which will immediately hand over to the bootloader on the ssd. This is a brief read-only access so the normal reliability issues shouldn't apply. Since the sd card is never mounted and thus invisible to homeassistant this will survive upgrades.

Just tried this with a fresh 5.10 image on the ssd but still no boot i'm affraid.

eketjall avatar Jan 16 '21 16:01 eketjall

Upgrade from 4.17 to 5.8 works and booted on my end using a Samsung 64GB USB flash drive on a Raspberry Pi 3+, so it seems this depends on the environment :-(

There could be several things which might be the culprit: Firmware, U-Boot or the new Linux Kernel 5.4. What would be interesting is to test different combinations of firmware/U-Boot with the 5.8 release: E.g. to test the old firmware, copy start.elf, fixup.dat and bootcode.bin from the first boot partition to the same place of a 5.8 USB drive. To test U-Boot from 4.17 copy u-boot.bin and boot.scr to the same place of a 5.8 USB drive.

When the screen turns black, it must be either the Raspberry Pi firmware or U-Boot which is causing issues. What would be interesting is to use the firmware

I've tried this in different combinations using boot files from 4.20 on an 5.10 image 1 - start.elf, fixup.dat and bootcode.bin 2 - u-boot.bin and boot.scr 3 - 1 and 2 at the same time.

Still no boot from ssd.

If all the boot files, incl overlays folder is used from 4.20 then it works.

eketjall avatar Jan 16 '21 18:01 eketjall