balena-os
balena-os copied to clipboard
balena-os
Allow USB boot on Raspberry Pi 3 boards
ResinOs already supports program_usb_boot_mode on Raspberry Pi 3 as stated on #283 but it want boot from USB.
The OTP memory in the SoC is programmed correctly when the device configuration variable RESIN_HOST_CONFIG_program_usb_boot_mode is set to 1, that is, the command "vcgencmd otp_dump | grep 17:" shows the expected value "3020000a".
I've tried both Etcher and Win32DiskImager to flash a ResinOS image (even tried with the latest 2.3.0) onto a Sandisk Cruzer Fit 16GB (reported to be compatible on https://www.raspberrypi.org/blog/pi-3-booting-part-i-usb-mass-storage-boot/) but the system cant' boot. Never tried to use "resin local flash" as shown on https://resinos.io/docs/architecture/ but I would expect the same results. I've also tried to flash NOOBS onto the same USB device and the system can boot!
Upon investigation I discovered that the cmdline.txt file within the resin-boot partition contains "root=/dev/mmcblk0p2". This value is going to work with memory cards only, not USB devices that are named /dev/sdaN. Changing that value to /dev/sda2 would allow ResinOS to boot! But as soon as the system is booted I can see that the /data partition has not been resized to take the whole space on the USB device. Maybe because of another reference to /dev/mmcblk0p2, I suppose.
I've googled a bit and I can say that se same problem (and solution) has been reported by different people on https://community.home-assistant.io/t/usb-boot-on-raspberry-pi-3/20358 when talking about Hass.io, the Resin-based version of Home Assistant by Pascal Vizeli (@pvizeli on GitHub).
I think persistent block device naming should be preferred. That is, something like "root=PARTUUID=xxx" (or even "root=PARTLABEL=xxx", if supported) should be used within cmdline.txt and devices should be reffered to using their UUID (or label, if supported). This should allow we to move from SD cards to USB devices. And I would the investigate in a USB failover solution, since we can connect more that one USB device to the RPi.
Don't know if this issue should go here or on https://github.com/resin-os/resin-raspberrypi , sorry.
@agherzan It gets a bit confusing, but this is something very different to what we're doing with amber-etcher-kernel ;-)
In amber-etcher-kernel we're using the usbboot mode of the BCM2835 and BCM2837 to have the Pi act as a USB slave (device) - this only works on Pis without an on-board USB hub (CM1, A, A+, CM3, PiZero) and is documented at https://www.raspberrypi.org/documentation/hardware/computemodule/cm-emmc-flashing.md and https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/device.md
The feature being requested here is about the Pi acting as a USB master (host), and booting from a USB memory stick instead of booting off the SD card - this only works on Pis based on the BCM2837 (3B and CM3) and is documented at https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/msd.md and https://www.raspberrypi.org/blog/pi-3-booting-part-i-usb-mass-storage-boot/
And there's a higher-level overview of how all the different boot modes "fit together" at https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/bootflow.md
@lurch is right.
Also, please note that RPi3 is able to boot via USB actually! But when the RPi3 reads the cmdline.txt it will hang on "root=/dev/mmcblk0p2". This is because "/dev/mmcblkNpN" is a standard naming for MMC/SD, while USB memory sticks would use "/dev/sdN"!
Now, if you just place the correct value within cmidline.txt (that is, "/dev/sda2" in place of "/dev/mmcblk0p2" the RPi3 would be able to complete the boot.
But you will soon find that the /data partition has not been resized! I suppose this is due to the fact that a script is somewhere trying to resize "/dev/mmcblk0p2" and not "/dev/sda2".
In the end, that's why I say that persistent block device naming should be preferred.
That is, something like "root=PARTUUID=xxx" (or even "root=PARTLABEL=xxx", if supported) should be used within cmdline.txt and this UUID (or label) should be used everywhere when needed to refer to the device.
I just had a quick look at how the Raspbian images provided by the RPF deal with this... they use root=PARTUUID=xxx in /boot/cmdline.txt, PARTUUID=xxx in /etc/fstab, and their partition-resize script ( https://github.com/RPi-Distro/raspi-config/blob/master/init_resize.sh ) uses findmnt to identify the partitions, and then updates both /etc/fstab and /boot/cmdline.txt post-resize with the disk's new PARTUUID values.
...and here's where the initial PARTUUID values get set in the image before it's been booted for the first time: https://github.com/RPi-Distro/pi-gen/blob/dev/export-image/03-set-partuuid/00-run.sh
EDIT: and the skip=440 bs=1 count=4 in there corresponds to the 32-bit disk signature stored in the https://en.wikipedia.org/wiki/Master_boot_record
And @lurch is right one more time! 😄
I've tried Raspbian before with success and I know it uses PARTUUID. PARTLABEL may be an alternative, but I don't know if it it supported within cmdline.txt.
I think that PARTUUID is the correct way to go.
Nope, I've not touched any of the ResinOS code - the only "work" I've done was the research that I documented in my above comments.
Don't know if/how I may help. Don't even know if anyone else is interested. But I am. :) Maybe just two actions have to be taken:
- Use PARTLABEL or even PARTUUID with "root=
" inside cmdline.txt - Modify the partition resizing script to work on PARTLABEL or PARTUUID
Maybe you know where this code resides. Don't know if this is Resin-related stuff or Yocto. Maybe with some hints and tips from you I may come back with a patch file. :)
I'm kind of interested. As I would like to boot from NFS. I do so on a couple of Raspian OSs I have an SD card with the FAT32 boot partition (and all othars, but I don't use them) After initial boot and start up I copy all partions to an NFS share, and then I edit the cmdline.txt in the FAT32 partion to boot from my NFS share. That way I avoid accessing the SD card except at first boot mooment. And that makes my SD cards live longer.
But I'm all new to ResinOS, and can't find any information if that is possible with ResinOS.
Did you find a solution for USB boot?
Again, don't know if/how I may help, but if someone can tell me where in the code the config.txt gets modified for the boot partition and the name of the script resizing the partition I can try to add real USB support. As said before, #283 is useless without #396.
@agherzan and @lurch do you think that I can try to contribute in some way?
IMHO #283 is useless without #396, but a lot of works seems to have been already done. I've also manifested my interested in Project Fin (pre-ordering some units right now) which uses the Compute Module, thus an eMMC and not a microSD, since the first seem to be a more reliable choice.
It seems that the whole Resin,io would have no problem running from either a microSD or a USB pen drive. Maybe we just have to use partition labels in place of device names both in the boot loader and in the partition resize script.
@xoul As far as I am aware the entire needed work for USB boot was already done. In the latest OS, all the hardcoded references to the block device name were replaced by labels. So at this point, I would expect that formatting a USB pen with an SD card image would just work. Have you tried that? If not, can you give it a go?
@agherzan I've tested both a Raspberry Pi 3 B and a 3 B+ devices and these are the results:
- Raspberry Pi 3 B+ does not boot via USB
- Raspberry Pi 3 B does boot via USB, but the /data filesystem does not get resized to occupy the whole space
Both tested multiple times with two different USB pen drives, one of which is a 16 GB SanDisk Cruzer Fit (know to be compatible, as shown on https://www.raspberrypi.org/blog/pi-3-booting-part-i-usb-mass-storage-boot/). Both prepared by setting RESIN_HOST_CONFIG_program_usb_boot_mode to 1 and tested with "vcgencmd otp_dump | grep 17:3020000a". To be honest, I forgot that RPi 3 B+ does not require special preparation to boot via USB: I'm trying to understand if having set program_usb_boot_mode may have caused problems.
But going back to the RPi 3 B, when booted from a 16 GB SanDisk Cruzer Fit, /data is only 1 GB, so I think that maybe the resize/rapartition/whatever script does not honour disk labels:
root@ec2c4f4:/usr/src/app# fdisk -l /dev/sda
Disk /dev/sda: 14.6 GiB, 15664676864 bytes, 30595072 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x6d750009
Device Boot Start End Sectors Size Id Type /dev/sda1 * 8192 90111 81920 40M e W95 FAT16 (LBA) /dev/sda2 90112 729087 638976 312M 83 Linux /dev/sda3 729088 1368063 638976 312M 83 Linux /dev/sda4 1368064 3530751 2162688 1G f W95 Ext'd (LBA) /dev/sda5 1376256 1417215 40960 20M 83 Linux /dev/sda6 1425408 3530751 2105344 1G 83 Linux
@agherzan again, I'm here to do some testing, if needed. Also, if you know of other issues on the subject, I would love to follow them.
I've found https://forums.resin.io/t/raspberry-to-run-resin-io-from-hard-disk/2413/17. It seems that the RPi can now boot via USB, but the resin-data partition is not resized.
I have the same problem. I am testing with CM 3. I tried to extend it directly with fdisk, but I could not do it.
root@6a6bb68:~# fdisk -l /dev/sda
Disk /dev/sda: 14.7 GiB, 15804137472 bytes, 30867456 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x56e5489f
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 8192 90111 81920 40M e W95 FAT16 (LBA)
/dev/sda2 90112 729087 638976 312M 83 Linux
/dev/sda3 729088 1368063 638976 312M 83 Linux
/dev/sda4 1368064 3530751 2162688 1G f W95 Ext'd (LBA)
/dev/sda5 1376256 1417215 40960 20M 83 Linux
/dev/sda6 1425408 3530751 2105344 1G 83 Linux
root@6a6bb68:~# systemctl status resin-filesystem-expand.service
● resin-filesystem-expand.service - Expand resin-data partition
Loaded: loaded (/lib/systemd/system/resin-filesystem-expand.service; enabled; vendor preset: enabled)
Active: active (exited) since Mon 2018-07-16 17:12:38 UTC; 1 months 1 days ago
Process: 604 ExecStart=/bin/sh /usr/bin/resin-filesystem-expand (code=exited, status=0/SUCCESS)
Main PID: 604 (code=exited, status=0/SUCCESS)
Jul 16 17:12:37 6a6bb68 sh[604]: Fix? yes
Jul 16 17:12:37 6a6bb68 sh[604]: Free inodes count wrong (94412, counted=101825).
Jul 16 17:12:37 6a6bb68 sh[604]: Fix? yes
Jul 16 17:12:37 6a6bb68 sh[604]: resin-data: ***** FILE SYSTEM WAS MODIFIED *****
Jul 16 17:12:37 6a6bb68 sh[604]: resin-data: 29247/131072 files (0.1% non-contiguous), 194974/262144 blocks
Jul 16 17:12:37 6a6bb68 sh[604]: [INFO] Expand ext4 filesystem on /dev/disk/by-label/resin-data...
Jul 16 17:12:37 6a6bb68 sh[604]: resize2fs 1.43.8 (1-Jan-2018)
Jul 16 17:12:38 6a6bb68 sh[604]: Resizing the filesystem on /dev/disk/by-label/resin-data to 263168 (4k) blocks.
Jul 16 17:12:38 6a6bb68 sh[604]: The filesystem on /dev/disk/by-label/resin-data is now 262144 (4k) blocks long.
Jul 16 17:12:38 6a6bb68 systemd[1]: Started Expand resin-data partition.
root@6a6bb68:~# resin-filesystem-expand
[INFO] Filesystem check on /dev/disk/by-label/resin-data...
e2fsck 1.43.8 (1-Jan-2018)
/dev/disk/by-label/resin-data is mounted.
e2fsck: Cannot continue, aborting.
[INFO] Expand ext4 filesystem on /dev/disk/by-label/resin-data...
resize2fs 1.43.8 (1-Jan-2018)
Filesystem at /dev/disk/by-label/resin-data is mounted on /mnt/data; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/disk/by-label/resin-data is now 263168 (4k) blocks long.
root@6a6bb68:~#
Hi @lurch and @agherzan! As you may see I had to convert @xoul into an organization, so I'm now reachable as @daghemo. As said on https://forums.resin.io/t/raspberry-to-run-resin-io-from-hard-disk/2413/37 I now have a console cable to investigate the problem at boot time on a Raspberry.
Hi @lurch and @agherzan! I'm ready for testing. Right now, Raspberry Pi devices can boot from USB, but the filesystem does not get resized so the whole stuff remains (almost) unusable.
Just as an update - there are instructions posted by @LucianBuzzo in the forums that allow for manual intervention that should get the device to a booting state.
I think https://github.com/balena-io/etcher/issues/1451 may also be of interest, as a future workaround.
Hi, I have a CM3 with 4GB eMMC and would like to be able to boot from USB stick memory, anyone was able to do so?
@IronFist16 it worked for some OS versions.
We have some work to do to make it work for recent OS. For more discussion, in chronological order https://github.com/balena-os/balena-raspberrypi/issues/341 https://github.com/balena-os/meta-balena/issues/1575 https://github.com/balena-os/balena-raspberrypi/pull/421#issuecomment-554486499
