PARTUUID Changes after Replacing OS and First Boot

[billsidea]

What is the expected behavior regarding PARTUUID value changes for the related partitions when using the "Replace" feature on the "Maintenance" menu? I had some challenges but was able to keep things working by manually changing "partition" values in the "installed_os.json" file. I'm just wondering if there is an easier way.

I started with a fresh install of PINN (PINN-LITE.img). I am using an Argon One case with a 250GB SATA M.2 V-NAND SSD (Samsung) which is exposed to the Pi as a USB drive.

The initial install included one Raspberry Pi OS and four "Project Space" placeholders. I then did some experimentation with replacing the various Project Spaces as well as replacing installed real OS instances.

In summary, I found that I had to change the PARTUUID references (in "installed_os.json") to "/dev/sdaXX" references. If I didn't do that, the OSs would keep working, but PINN lost track of the partitions (and thus I could not do anything again with those OS instances in PINN like replace them again or use the PINN "Edit Config" feature).

The thing that surprised me most is the changing of the PARTUUID only after the initial boot of the new OS. That basically killed PINN's ability to talk to it, I think.

Am I doing something improperly with the "Replace" feature or is this just a limitation?

I have some step-by-step details below if that helps. I can also provide images of each of these steps if you'd like.

Thanks!

Taking a closer look at one example, these were the steps:

1. Replaced an existing Raspberry Pi OS with a new Raspberry Pi OS using the "Replace" feature.
2. Before booting the new OS, I confirmed that PARTUUID values were all in sync between "installed_os.json", "/etc/fstab" on the related root partition, and in the PINN Config editor (cmdline.txt "root" value). These values were represented as PARTUUID (e.g. "PARTUUID=3418e696-0c" and "PARTUUID=3418e696-0d").
3. Booted to the new OS. It worked fine. I then rebooted and went back into PINN.
4. Tried to do an "Edit Config" on the OS in PINN, but it resulted in an "Error mounting boot partition" error.
5. Looked at the "/etc/fstab" file on the related root partition and saw that the values had changed to another PARTUUID value (e.g. "PARTUUID=1d47c601-0c" and "PARTUUID=1d47c601-0d").
6. Looked at the "installed_os.json" file and saw that those were still the original values (e.g. "PARTUUID=3418e696-0c" and "PARTUUID=3418e696-0d").
7. Updated the "installed_os.json" file to specify "/dev/sda12" and "/dev/sda13" instead.
8. Rebooted and went back into PINN.
9. Tried to do an "Edit Config" on the OS in PINN and it worked successfully. The "cmdline.txt" showed "root=PARTUUID=1d47c601-0d".
10. Initiated the "Re-run partition_setup.sh" option.
11. Went back into "Edit Config" and saw that "cmdline.txt" had been updated to show "root=/dev/sda13".

[procount]

Thanks for your detailed report. This is quite strange as one of the main features of PINN is for it to manage these PARTUUID values for you, and it has been working well for many years. This is the first report of such an issue, so something is wrong somewhere. It sounds like the OS is modifying the PARTUUID after installation. This could be due to a number of things, so I will need to investigate.

You left out some details: which model of raspberry Pi are you using? Which of the 6 variants of the Raspberry Pi OS were you installing (32/64, full/standard/lite)?

[lurch]

It sounds like the OS is modifying the PARTUUID after installation.

When you use e.g. Raspberry Pi Imager to write a Raspberry Pi OS image onto an SD card, the PARTUUIDs end up being those in the image file you downloaded. UUIDs are supposed to be Universally Unique IDentifiers, but they won't be unique if every image has the same PARTUUIDs, and so Raspberry Pi OS generates new unique PARTUUIDs on the first boot. https://github.com/RPi-Distro/raspberrypi-sys-mods/blob/master/usr/lib/raspberrypi-sys-mods/firstboot#L63

it has been working well for many years

Maybe it's now "broken" because NOOBS is no longer officially supported? :man_shrugging:

[procount]

Could be. I recall there was special noobs detection code at some point. I used to remove/adapt the firstrun script to prevent the unnecessary root partition expansion, but leaving it didn't seem to make much difference previously. Maybe the noobs part has been removed now and I need to do some adaptions.

[billsidea]

You left out some details: which model of raspberry Pi are you using? Which of the 6 variants of the Raspberry Pi OS were you installing (32/64, full/standard/lite)?

This is on a Raspberry Pi 4 Model B Rev 1.4 with 8GB of RAM. My detailed example was for the "raspios_armhf" (A port of Debian with the Raspberry Pi Desktop) image, but I also experienced this issue with the "twister" (A themed Raspberry Pi OS Desktop merged with gaming v2.1.2) image. The PINN version is v3.8.7 (Built: May 6 2023).

[billsidea]

There is another point worth mentioning about when you're in this situation where you've replaced an OS using PINN's "Replace" feature but have not yet worked around the issue (by manually updating the "installed_os.json" file). The OS will keep working and you can still boot into it. However, if you try to do another "Replace" on that instance, you will get this type of error: "Cannot Reinstall/Replace raspios_armhf. Partition not big enough for new image." I believe it is essentially the same technical issue as I described above, but the message might take you in another direction for troubleshooting. I think I ran across some other issues/posts where people were seeing this message. I just wanted to point that out in case it may help with any other issues you may be fielding.

[procount]

Yes, that's certainly relevant, thanks. Does "re-running the partition_setup.sh script" from the fix up menu sort this out, without you having to mess with installed_os.json?

[procount]

@Lurch - The firstboot script still checks for the presence of NOOBS, but doesn't take this into account before calling fix_partuuid. Looks like this has been an issue since 20220901 release for rpi_imager customisations. I have raised an issue for XECDesign to consider.

[billsidea]

Does "re-running the partition_setup.sh script" from the fix up menu sort this out, without you having to mess with installed_os.json?

Unfortunately, no. In this scenario, running that script results in the following:

Checking for partition_setup.sh scripts...
Executing /settings/os/raspios_armhf/partition_setup.sh
part1= id1=LABEL=RECOVERY
SETTINGS
boot
root
boot64
root64
boot640
root640
boot1
root0
boot0
twister part2= id2=LABEL=RECOVERY
SETTINGS
boot
root
boot64
root64
boot640
root640
boot1
root0
boot0
twister
raspios_armhf: Error executing partition setup script

raspios_armhf partions_setup.sh script error

DONE - some setup script errors detected

[billsidea]

FYI, I did another test of the same scenario, but I did the workaround slightly differently. Instead of updating "installed_os.json" with the "/dev/sdaXX" values, I updated "installed_os.json" with the newly-regenerated PARTUUID values (as taken from "/etc/fstab"). This also appears to fix the issue. After doing that, I can once again use the PINN features for that OS (e.g. "Edit Config"). If I then do the "Re-run partition_setup.sh" from the fix menu, it works and updates "cmdline.txt" and "/etc/fstab" with the "/dev/sdaXX" values. However, the "installed_os.json" file continues to have the (correct) PARTUUID values and everything works fine.

[lurch]

Just as an extra bit of info as to why these PARTUUID values are being used instead of always using /dev/sdaXX values... (in case you're not already aware of this) If you only have one USB drive plugged in (i.e. your SSD) then it'll always appear as /dev/sda. But if you plugged in another USB drive (e.g. a USB flash drive) and then rebooted, and if that flash drive initialised faster than your SSD, then your USB flash drive would be /dev/sda but your SSD would now be /dev/sdb and any references to a /dev/sdaXX partition wouldn't work :wink: In the early days of NOOBS (which PINN has evolved from) this wasn't a problem, because the Raspberry Pi was only able to boot from SD card, and that always appears as /dev/mmcblk0 regardless of what you've got plugged into your USB ports. Booting directly from USB was IIRC a feature added by the Pi 3.

[billsidea]

Thanks for that info, @lurch. That makes sense. Although convenient to reference the /dev/sdaXX values because they are more stable than the PARTUUID values in one sense, they are not as stable in a "multi-USB-drive" environment. My testing does show that referencing the "PARTUUID" values does work (once manually fixed in "installed_os.json" after first boot), but I'm just not sure how PINN would be able to sense the PARTUUID change upon first boot of the OS. Maybe theres a way to alter that "Re-run partition_setup.sh" feature to somehow try to parse that OS's "/etc/fstab" file (and then use those PARTUUID values rather than the /dev/sdaXX values going forward).

[procount]

We just need to stop Raspios (and others based on it) from modifying the diskid when in a multi-boot environment. I'm working on it.....

[lurch]

somehow try to parse that OS's "/etc/fstab" file

No need to mount the rootfs partition and try parsing the enclosed /etc/fstab, you can just read the PARTUUID directly from the partitions (it's part of the metadata / header-info).

[billsidea]

We just need to stop Raspios (and others based on it) from modifying the diskid when in a multi-boot environment.

Out of curiosity, if those changes were made, if a PINN user installed multiple copies of the same OS image on the same drive, would there be a PARTUUID clash that would potentially break things? I'm willing to help test that use case if you think it's valid and have any concerns about it.

[procount]

Out of curiosity, if those changes were made, if a PINN user installed multiple copies of the same OS image on the same drive, would there be a PARTUUID clash that would potentially break things?

No because the PARTUUID is DISKID-PARTNO, which will be unique.

[mkreisl]

I also have the same problem and have been annoyed about it for days. I am using the beta on a Raspberry Pi 5. However, I have the impression that this PARTUUID only changes after I have booted ALL installed operating systems at least once.

In my opinion, the use of the PARTUUID is therefore complete nonsense and should finally be changed as quickly as possible instead of constantly adding gimmicks that hardly anyone needs anyway. I'm sorry, but user-friendliness is something else!

I wonder why the PARTUUID changes anyway, as it is derived from the disk identifier. Change it to UUID and the problem would be solved!

[procount]

@mkreisl - Which OS are you installing (exact name, version, release date would help)

[procount]

I don't think switching to UUID will totally help. The problem was that everyone who downloads the standard image would have the same PARTUUIDs or UUIDs. So to make these unique, a startup script in the OS is run to alter the PARTUUID during first boot. This is fine when it is the sole OS on the drive, but under a multi-boot scenario where PINN records the PARTUUIDs to know which partition to boot, changing the PARTUUID after the fact is going to cause problems. Especially since changing the diskid affects all the PARTUUIDS, so all of theOSes will be affected. Changing to UUID and adjusting it on first boot would just cause that OS to be affected, thus lessening the impact, but it would still cause a problem for that OS.

For Raspios, this has already been fixed since it no longer changes the PARTUUID if it detects it is installed under PINN, since PINN will have already created a unique diskid for the drive, so all PARTUUIDs will be unique by design.

If you are having problems with another OS, then any script needs to be modified to not change diskids if it detects NOOBS/PINN, or the partition_setup.sh script can be modified to remove or adapt such a script.

[mkreisl]

@mkreisl - Which OS are you installing (exact name, version, release date would help)

I had tried several times, I will now describe my last attempt here: The hardware I used is a Raspberry Pi 5/8GB with external 250GB USB3 disk, no sd card.

I had installed XBian in the 32-bit and 64-bit version, as well as ubuntu2310 and raspios_arm64 So that I could install my XBian versions at all, I had changed the cmdline.txt as follows:

runinstaller quiet ramdisk_size=65536 root=/dev/ram0 init=/init vt.cur_default=1 elevator=deadline repo_list=http://sourceforge.net/projects/xbian/files/noobs/repo_list.json ssh loglevel=2 sdhci.debug_quirks2=4

I have also activated ssh, otherwise everything corresponds to the original pinn-lite.zip from the beta folder

After all 4 operating systems finally landed successfully on the disk, I started Pi-OS first and set it up briefly. after this was done, I booted it again sucessfully.

Then I started the 32-bit XBian and then the 64-bit XBian. I proceeded in the same way as with Pi-OS, set up briefly and booted again. In between I booted Pi-OS again and again to check whether the PARTUUID's had changed, they hadn't so far

The last thing to go was Ubuntu. After I had set everything up so far, I carried out the update offered there without rebooting and then rebooted immediately. But it didn't want to boot anymore, it ended up in the initramfs command line, ext4 couldn't be mounted anymore because something went wrong with the fs. In any case it could be repaired with e2fsck. but I had done that on another pc.

Anyway, the disk identifier had now changed and thus the PARTUUIDs, so the end of booting for the systems that use the PARTUUIDs

[mkreisl]

I don't think switching to UUID will totally help. The problem was that everyone who downloads the standard image would have the same PARTUUIDs or UUIDs.

I really don't understand what you mean. It's about the newly installed systems by PINN, for this purpose 2 partitions are created for each operating system and thus these two partitions receive a unique UUID, that's the point of the UUID, that it is really unique. I also don't know how to change the disk identifiers and thus the PARTUUIDs by normal means, but obviously this happens as you have described.

[mkreisl]

I have added an extension to partition_setup.sh that replaces the PARTUUID's passed with the UUID.

convert_partuuid() {
    uuid=''
    if echo "$1" | grep -q 'PARTUUID'; then
        p=$(echo $1 | awk -F = '{print $2}')
        [ -n "$p" ] && uuid=$(blkid | grep "$p" | awk -F UUID= '{print $2}' | tr -d \" | awk '{print $1}')
    fi
    [ -n "$uuid" ] && echo "UUID=$uuid" || echo "$1"
}

part1=$(convert_partuuid $part1)
part2=$(convert_partuuid $part2)

Then whoever wants to can change the disk identifier and thus the PARTUUIDs, the installed system then cares very little about this. Since then there has been no problem. But of course you can't do that with other people's things

[procount]

Just to be clear:

1. There is no reason for the PARTUUIDs to change, and they shouldn't.
2. PINN is not changing the PARTUUIDs
3. Most likely, this is due to the initial scripts in Raspios., although as far as I am aware, that bug has been fixed. See https://github.com/RPi-Distro/raspberrypi-sys-mods/blob/3af395f60cfc5061d72ea79f3d33bf0ccdac29a8/usr/lib/raspberrypi-sys-mods/firstboot#L25, which is why I was asking for the specific version of the OS.

Yes you are right, UUIDS (and PARTUUIDs) should be unique. The problem is that an OS is normally distributed as an IMAGE file, which contains the UUIDS/PARTUUIDS/DISKIDs. So every installation is essentially a clone. Normally this is ok until you put 2 or more of those cloned partitions on the same system. There were some instances of this, so RPT chose to change the diskid during the initial configuration of the OS. This is fine on a single OS drive, but catastrophic in a multi-boot environment because it changes the PARTUUID of all the other OSes.

So, you can see in the link above that they now check if the boot partition is not 1 (in which case it is running under NOOBS/PINN) and then they do not fixup the partuuid. So it should not happen anymore. As it is still happening to you we need to debug that.

I don't think it is related to Ubuntu because it references its partitions via labels (not ideal for other reasons) and so would have no need to change the disk id.

[mkreisl]

I don't think it's just due to a script running amok in Pi-OS, because I made my first attempts (it must have been at least 4 attempts or more) all without Pi-OS, so only XBian and Ubuntu. It also occurred there. I also suspected XBian, because when the root partition is at the end, it shrinks the root partition and appends a swap partition. but I checked the script and couldn't see that this changes the disk identifier. As I said, I don't know how this is supposed to work. that would make the whole thing absurd anyway. The use of fdisk to change this has already been investigated, but I don't have an fdisk that allows you to change this id using the i command

I don't think it is related to Ubuntu because it references its partitions via labels (not ideal for other reasons) and so would have no need to change the disk id.

I agree with you

[procount]

I will try to follow your installation steps to see if I can replicate your issue. It may take a little while....

[procount]

Re-reading your procedure, maybe it was the ubuntu update that killed it? Was it version 23.10?

[mkreisl]

Re-reading your procedure, maybe it was the ubuntu update that killed it? Was it version 23.10?

Yes

[mkreisl]

I have now carried out the procedure again, but after starting Pi-OS 64-bit and its configuration, update with subsequent reboot, during which I checked the PARTUUID, I immediately started XBian 64-bit, which, since it was installed at the end of the disk, created a swap partition.

I then checked the PARTUUID without rebooting and it had changed.

So now again, the question is, how can I change the PARTUUID? Tomorrow I will carry out a few more tests, but I stick to my opinion: the use of PARTUUID's is in my opinion absolute nonsense

[procount]

Has Xbian always created such a swap partition? That's not really compatible with the way PINN works, although it shouldn't make a difference if it only happens if it is at the end of the disk. It might be better to explicitly allocate a swap partition for it no matter where it is located.

[procount]

So now again, the question is, how can I change the PARTUUID?

You change the PARTUUID by changing the diskid using fdisk or similar. PARTUUID = diskid-partitionNumber

[mkreisl]

Yes, this has always been the case. But only if the XBian root partition is at the end of the disk, otherwise it can't work. Tomorrow I will suppress creation of the swap partition, this can be done simply by deleting a parameter in cmdline.txt. Let's see what happens then.