Utilities System Update utility

Write a System Update utility that installs the update into a new Boot Environment.

Along the lines of

 pkg upgrade --fetch-only -y
 beadm list
 beadm create c572e
 beadm mount c572e /media/c572e
 chroot /media/c572e
  pkg upgrade -y
  pkg upgrade -r FreeBSD -y
  exit
 beadm list
 beadm umount c572e

Thanks @grahamperrin

We could then make the next boot go into the new Boot Environment, and ask the user there whether all subsequent boots should continue to go there. Or something like that.

Jan 17 '21 21:01 probonopd

Thanks, that's from my quick and dirty record of a first attempt https://pastebin.com/ppT7LfVT where (amongst other things) the pkg upgrade -y failed due to poudriere-related files being not found in the chroot(8) environment.

I took my hints from Upgrading from FreeBSD 9.3 to FreeBSD 10.1 using beadm and freebsd-update – Dan Langille's Other Diary (2015-03-13).

I see options such as --chroot and --rootdir in pkg(8), my first experiment with --chroot failed (through me not yet having a full understanding of the manual page). I guess that using one or both of those options in combination with (or in lieu of) chroot(8) will allow things to work with e.g. my poudriere repository, I simply haven't had time to digest the man pages.

For the pkg stuff above, the phrase System Update is potentially misleading in that base (the system) is not yet packaged. So better name the utility Software Update or something like that.

A System Update to the FreeBSD -RELEASE base of helloSystem should use freebsd-update(8).

A System Update to the FreeBSD -CURRENT base of helloSystem might be out of scope (without PkgBase).

FreeBSD upgraded, no `/dev/null` after `chroot`

grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % sudo pkg install beadm
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        beadm: 1.3.2

Number of packages to be installed: 1

10 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching beadm-1.3.2.txz: 100%   10 KiB  10.1kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing beadm-1.3.2...
[1/1] Extracting beadm-1.3.2: 100%
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % pkg info -x mysql
mysql57-client-5.7.32
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % beadm list
BE      Active Mountpoint  Space Created
default NR     /            4.4G 2021-02-04 14:49
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % freebsd-version -kru
12.2-RELEASE-p3
12.2-RELEASE-p3
12.2-RELEASE-p3
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % sudo beadm create 12.2-RELEASE-p3
Created successfully
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % sudo beadm mount 12.2-RELEASE-p3 /media/12.2-RELEASE-p3
Mounted successfully on '/media/12.2-RELEASE-p3'
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % chroot /media/12.2-RELEASE-p3
chroot: /media/12.2-RELEASE-p3: Operation not permitted
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % sudo chroot /media/12.2-RELEASE-p3
root@mowa219-gjp4-vm-hellosystem-040-0d26:/ # sudo pkg upgrade -y
sudo: account validation failure, is your account locked?
sudo: a password is required
root@mowa219-gjp4-vm-hellosystem-040-0d26:/ # whoami
root
root@mowa219-gjp4-vm-hellosystem-040-0d26:/ # pkg upgrade -y
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking for upgrades (18 candidates): 100%
Processing candidates (18 candidates): 100%
The following 16 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        abseil: 20200923.2_1 -> 20200923.3
        drm-fbsd12.0-kmod: 4.16.g20201016 -> 4.16.g20201016_1
        libunwind: 20200331_1 -> 20201110
        libx264: 0.161.3020 -> 0.161.3039
        mysql57-client: 5.7.32 -> 5.7.32_1
        opencv-core: 3.4.1_39 -> 3.4.1_40
        pciids: 20201127 -> 20201231
        protobuf: 3.13.0,1 -> 3.14.0,1
        readline: 8.0.4_1 -> 8.1.0
        sudo: 1.9.5p1 -> 1.9.5p2
        wayland: 1.18.0_4 -> 1.19.0
        xorg-server: 1.20.9_1,1 -> 1.20.9_3,1
        zstd: 1.4.5_1 -> 1.4.8

Installed packages to be REINSTALLED:
        freeglut-3.0.0_2 (direct dependency changed: libXxf86vm)
        libdvdread-6.1.0 (options changed)
        virtualbox-ose-additions-5.2.44_3 (options changed)

Number of packages to be upgraded: 13
Number of packages to be reinstalled: 3

15 MiB to be downloaded.
[1/16] Fetching zstd-1.4.8.txz: 100%  504 KiB 516.6kB/s    00:01    
[2/16] Fetching xorg-server-1.20.9_3,1.txz: 100%    1 MiB   1.5MB/s    00:01    
[3/16] Fetching wayland-1.19.0.txz: 100%  120 KiB 123.0kB/s    00:01    
[4/16] Fetching virtualbox-ose-additions-5.2.44_3.txz: 100%  682 KiB 698.7kB/s    00:01    
[5/16] Fetching sudo-1.9.5p2.txz: 100%  942 KiB 964.9kB/s    00:01    
[6/16] Fetching readline-8.1.0.txz: 100%  362 KiB 370.5kB/s    00:01    
[7/16] Fetching protobuf-3.14.0,1.txz: 100%    3 MiB   2.9MB/s    00:01    
[8/16] Fetching pciids-20201231.txz: 100%  216 KiB 220.7kB/s    00:01    
[9/16] Fetching opencv-core-3.4.1_40.txz: 100%    2 MiB   2.2MB/s    00:01    
[10/16] Fetching mysql57-client-5.7.32_1.txz: 100%    2 MiB   1.9MB/s    00:01    
[11/16] Fetching libx264-0.161.3039.txz: 100%  678 KiB 694.2kB/s    00:01    
[12/16] Fetching libunwind-20201110.txz: 100%  128 KiB 131.2kB/s    00:01    
[13/16] Fetching libdvdread-6.1.0.txz: 100%  121 KiB 123.4kB/s    00:01    
[14/16] Fetching freeglut-3.0.0_2.txz: 100%  223 KiB 228.1kB/s    00:01    
[15/16] Fetching drm-fbsd12.0-kmod-4.16.g20201016_1.txz: 100%    2 MiB   2.1MB/s    00:01    
[16/16] Fetching abseil-20200923.3.txz: 100%  798 KiB 817.1kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/16] Upgrading pciids from 20201127 to 20201231...
[1/16] Extracting pciids-20201231: 100%
[2/16] Upgrading readline from 8.0.4_1 to 8.1.0...
[2/16] Extracting readline-8.1.0: 100%
pkg: Cannot open /dev/null:No such file or directory
[3/16] Upgrading zstd from 1.4.5_1 to 1.4.8...
[3/16] Extracting zstd-1.4.8: 100%
pkg: Cannot open /dev/null:No such file or directory
[4/16] Upgrading wayland from 1.18.0_4 to 1.19.0...
[4/16] Extracting wayland-1.19.0: 100%
pkg: Cannot open /dev/null:No such file or directory
[5/16] Upgrading libunwind from 20200331_1 to 20201110...
[5/16] Extracting libunwind-20201110: 100%
pkg: Cannot open /dev/null:No such file or directory
[6/16] Upgrading protobuf from 3.13.0,1 to 3.14.0,1...
[6/16] Extracting protobuf-3.14.0,1: 100%
pkg: Cannot open /dev/null:No such file or directory
[7/16] Upgrading xorg-server from 1.20.9_1,1 to 1.20.9_3,1...
pkg: Cannot open /dev/null:No such file or directory
root@mowa219-gjp4-vm-hellosystem-040-0d26:/ # file /dev/null
/dev/null: cannot open `/dev/null' (No such file or directory)
root@mowa219-gjp4-vm-hellosystem-040-0d26:/ # exit
exit
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % file /dev/null
/dev/null: character special (0/19)
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ % sudo beadm umount 12.2-RELEASE-p3 
Unmounted successfully
grahamperrin@mowa219-gjp4-vm-hellosystem-040-0d26:~ %

Re: https://markmail.org/message/ediqqmqrogbv6fum#query:+page:1+mid:ediqqmqrogbv6fum+state:results I assume that

FreeBSD 12 requires an extra step to make /dev/null available in the chroot environment
FreeBSD 13 and -CURRENT include some magic, to automatically fulfil the /dev/null requirement.

Feb 06 '21 13:02 grahamperrin

https://twitter.com/vermaden/status/1360562154836004869 describes upgrading from 12 to 13 as:

beadm create 13 beadm mount 13 chroot /var/tmp/BE-13.z4IanfZt mount -t devfs devfs /dev freebsd-update upgrade -r 13.0-BETA2 freebsd-update install freebsd-update install pkg upg freebsd-update install beadm activate 13 reboot

Feb 13 '21 12:02 probonopd

Thanks, that's useful.

https://www.freebsd.org/cgi/man.cgi?query=pkg-alias(8) upg is not an alias. Neither does it appear here:

https://www.freebsd.org/cgi/man.cgi?query=pkg-upgrade(8)
https://www.freebsd.org/cgi/man.cgi?query=pkg(8)

@vermaden please, where did you learn the upg abbreviation?

If the second run of freebsd-update install is for world, then what is the third?

Feb 16 '21 07:02 grahamperrin

@probonopd ~~for an upgrade with 0.4.0 (0D26) you'll not need the devfs line. So,~~ for example:

🚫 longhand – draft – I should advise users to not yet follow this routine in its entirety (note, any yellow alert below) …

with a clean installation of either 0.4.0 (0D26) or 0.5.0 (0E4) from hello-0.5.0_0E4-FreeBSD-12.1-amd64.iso
sudo -u root sh
observe the lecture-related hint
enter the password
sed -i -e 's|release_2|latest|g' /etc/pkg/FreeBSD.conf
pkg lock hello
y
pkg install beadm && mkdir /tmp/upgrade-freebsd
beadm create 13.0-BETA2-hs-iso-115 && beadm mount 13.0-BETA2-hs-iso-115 /tmp/upgrade-freebsd && chroot /tmp/upgrade-freebsd
mount -t devfs devfs /dev
freebsd-update upgrade -r 13.0-BETA2
freebsd-update install
exit
umount /tmp/upgrade-freebsd/dev
beadm umount 13.0-BETA2-hs-iso-115 && beadm activate 13.0-BETA2-hs-iso-115
exit
for the activated boot environment to become effective, restart helloSystem
Control-Alt-F2 switch to ttyv0
login at the console – do not attempt to use the desktop environment
sudo -u root sh
freebsd-update install
pkg bootstrap -f
pkg upgrade
freebsd-update install
exit
shutdown -r now

… then ⚠ https://github.com/helloSystem/ISO/issues/136 check whether Falkon is installable

with steps very similar to what's drafted above, it is not installable – as far as I can tell, still blocked by #115 so I begin to wonder whether (below, https://github.com/helloSystem/Utilities/issues/33#issuecomment-780083496) sudo -u root sh can be a workaround (to problems that may be associated with a locked root user).

Feb 16 '21 07:02 grahamperrin

https://cgit.freebsd.org/src/commit/usr.sbin/freebsd-update/freebsd-update.sh?id=48ffe56ac5b7adb5b851d32be12b2ec0f13705a4 (2006-08-31) ▶ https://cgit.freebsd.org/src/tree/usr.sbin/freebsd-update/freebsd-update.sh?id=48ffe56ac5b7adb5b851d32be12b2ec0f13705a4#n553

# Check that we are root.  All sorts of things won't work otherwise.

– and from BSD Server Maintenance - iXsystems, Inc. - Enterprise Storage & Servers (2013-09-30):

… When updating the system, type:

freebsd-update fetch; freebsd-update install

This will update the list of available software and then update the software. You must be root to apply such updates. …

– and so on.

https://cgit.freebsd.org/src/tree/usr.sbin/freebsd-update/freebsd-update.sh?id=48ffe56ac5b7adb5b851d32be12b2ec0f13705a4#n555

	echo "You must be root to run this."

– wonder why I never heard that echoed in the context of https://github.com/helloSystem/ISO/issues/115 …

freebsd-update(8) without an echo of "You must be root to run this."

https://lists.freebsd.org/pipermail/freebsd-current/2021-February/078833.html

Feb 16 '21 07:02 grahamperrin

@vermaden please, where did you learn the upg abbreviation?

Do not remember. Seriously :)

But pkg autor works the same for autoremove.

Probably a habit after long use of IBM TSM - Tivoli Storage Manager (now known as Spectrum Protect). Their command line also needed only that much part of command that was unique.

If the second run of freebsd-update install is for world, then what is the third?

After executing freebsd-update install the freebsd-update writes this at the end of 'world' install process:

Completing this upgrade requires removing old shared object files.
Please rebuild all installed 3rd party software (e.g., programs
installed from the ports tree) and then run "/usr/sbin/freebsd-update install"
again to finish installing updates.

So the third freebsd-update install is for 'removing old shared object files'.

for an upgrade with 0.4.0 (0D26) you'll not need the devfs line.

Not sure about helloSystem upgrade but when I first started 12.2 ==> 13.0 upgrade without /dev mounted in the chroot then the freebsd-update exited after installing files with information that /dev/tty is not available and its not possible to interactively 'work' on the configuration files differences like /etc/fstab or /etc/sysctl.conf files.

Regards.

Feb 16 '21 09:02 vermaden

From https://github.com/helloSystem/ISO/issues/138#issuecomment-778615459

… root user; … Mac. …

When I last checked, updates/upgrades to Mac OS X (familiarly macOS) involved booting an environment that does allow the root user.

So:

if (with helloSystem denial of root) https://github.com/helloSystem/Utilities/issues/33#issuecomment-779647953 above can not usefully involve chroot(8)
then maybe First Aid https://github.com/helloSystem/hello/issues/90#issuecomment-779701997

– however goals such as First Aid and Storage utilities might be mid- to long-term.

IMHO a more immediate priority should be to:

allow users of FreeBSD to update/upgrade the system using normal methods; and
allow these methods to be as close as possible to frictionless.

Thanks

(To do: link to a non-official shortlist of vulnerabilities.)

Feb 16 '21 09:02 grahamperrin

https://github.com/helloSystem/Utilities/issues/33#issuecomment-779700378 thanks @vermaden

`/dev`

~~Installed helloSystem 0.4.0 (0D26) mount point /dev has the required file system.~~

Old files

That reminds me,

root@mowa219-gjp4-8570p:~ # cd /usr/src && make -DBATCH_DELETE_OLD_FILES delete-old
>>> Removing old files (only deletes safe to delete libs)
/usr/sbin/fmtree
/usr/lib/debug/usr/sbin/fmtree.debug
/usr/share/man/man8/fmtree.8.gz
>>> Old files removed
>>> Removing old directories
>>> Old directories removed
To remove old libraries run 'make delete-old-libs'.
root@mowa219-gjp4-8570p:/usr/src #

– and

I do not make delete-old-libs
(off-topic from helloSystem) I can't easily recall what I have installed from ports
(OT) I do update (from source) my jail for poudriere, but I have never given thought to deletion of old files within the jail (does it take care of itself?) … maybe saner to simply create from source.

Feb 16 '21 09:02 grahamperrin

https://github.com/helloSystem/docs/pull/17

Feb 16 '21 18:02 grahamperrin

Taking a hint from https://pastebin.com/vSPb8Pxg (thanks to Remilia in freenode):

sudo -u root sh

– I'm trying, repeatedly, to fix the routine at https://github.com/helloSystem/Utilities/issues/33#issuecomment-779647953 above to work with root-locked helloSystem. So far, without success :-(

Feb 16 '21 20:02 grahamperrin

We could also take an entirely different approach to "updating", e.g., installing from a new Live ISO while keeping parts of the old system around, similar to what the Mac does... or use an A/B partitioning scheme, similar to what ChromeOS does, or...

Feb 16 '21 20:02 probonopd

… installing from a new Live ISO while keeping parts of the old system …

The parts gained from an ISO might as effectively be gained with a PkgBase approach. I think first of @igalic https://alpha.pkgbase.live/

Feb 16 '21 20:02 grahamperrin

Not really: Integration and end-to-end tests can be run on an ISO, but not on an upgraded system. At least not easily.

Feb 16 '21 21:02 probonopd

Integration

Way beyond me but 👍

end-to-end tests

Is an intention, to verify (with checksums) that what's installed is as it should be?

Feb 16 '21 21:02 grahamperrin

@vermaden

https://github.com/helloSystem/Utilities/issues/33#issuecomment-779711271 my bad,

Installed helloSystem 0.4.0 (0D26) mount point /dev has the required file system.

That was, not taking into account the chroot step. So, yes, mount -t devfs devfs /dev is a necessary next step.

Feb 16 '21 21:02 grahamperrin

#33 (comment) thanks @vermaden

`/dev`

Installed helloSystem 0.4.0 (0D26) mount point /dev has the required file system.

Old files

That reminds me,

root@mowa219-gjp4-8570p:~ # cd /usr/src && make -DBATCH_DELETE_OLD_FILES delete-old
>>> Removing old files (only deletes safe to delete libs)
/usr/sbin/fmtree
/usr/lib/debug/usr/sbin/fmtree.debug
/usr/share/man/man8/fmtree.8.gz
>>> Old files removed
>>> Removing old directories
>>> Old directories removed
To remove old libraries run 'make delete-old-libs'.
root@mowa219-gjp4-8570p:/usr/src #

– and

* I **do not** `make delete-old-libs`

* (off-topic from helloSystem) I can't easily recall what I have installed from ports

* (OT) I do _update_ (from source) my jail for poudriere, but I have never given thought to deletion of old files within the jail (does it take care of itself?) … maybe saner to simply _create_ from source.

Yes, this is that step.

Feb 16 '21 22:02 vermaden

@vermaden FYI https://pastebin.com/crR6WFCG – first observed whilst testing an upgrade with helloSystem, it's reproducible with a clean installation of FreeBSD ~~12.1-STABLE~~ 12.1-RELEASE (without helloSystem) as a starting point.

If I (exit the chroot environment etc. and) restart the system before the second run of freebsd-update install then there are no such bad system calls (and /lib/libmd.so.6 is not empty).

Feb 16 '21 23:02 grahamperrin

just one small note re beadm: if you replace it with bectl(8), which is in base, that's one less thing to install

Feb 17 '21 12:02 igalic

@vermaden FYI https://pastebin.com/crR6WFCG – first observed whilst testing an upgrade with helloSystem, it's reproducible with a clean installation of FreeBSD 12.1-STABLE (without helloSystem as a starting point.

If I (exit the chroot environment etc. and) restart the system before the second run of freebsd-update install then there are no such bad system calls (and /lib/libmd.so.6 is not empty).

If moving from 12.x to 13.x its best (and probably stated in the FreeBSD Handbook) to first upgrade 12.1 to 12.2 (or other latest 12.x version at the moment) and then do the major upgrade to 13.x.

Regards.

Feb 17 '21 12:02 vermaden

The more I think about it, I come to the conclusion that I am really more in favor of a "clean install" at least for major revisions/releases that would work roughly like this:

User downloads the latest ISO
In a running helloSystem, user double-clicks the ISO
User double-clicks "Install FreeBSD", the usual installer opens
The disk from which helloSystem is currently booted is shown
If the user selects that disk, then the following happens:
- A new jail/chroot is created that is also a boot environment
- helloSystem is installed to that jail/chroot
- /home is the same for the old and the new boot environment (as far as I remember this is even the default)
- /Applications may need some special treatment (tbd)
The new boot environment is selected for the next boot (only)
Only if we reach the full graphical desktop in the next boot, then set the new boot environment permanently

Advantages:

"Clean install" = predictable results
ISO can be tested beforehand
More familiar process to Mac users, but using ZFS power

For minor (security) updates, we could still use a more traditional approach.

What do you think?

Feb 17 '21 17:02 probonopd

Agreed.

Feb 17 '21 18:02 vermaden

Thanks,

https://github.com/helloSystem/Utilities/issues/33#issuecomment-780522191

… first upgrade 12.1 to 12.2 (or other latest 12.x version at the moment) and then do the major upgrade to 13.x. …

☑ that's sane,

… probably stated in the FreeBSD Handbook) …

– unless I'm missing something, it's not.

https://docs.freebsd.org/en/books/handbook/cutting-edge/#freebsdupdate-upgrade

Feb 18 '21 00:02 grahamperrin

https://github.com/helloSystem/Utilities/issues/33#issuecomment-778611679 @vermaden with the chroot approach, there is (I think) an extremely slim possibility of shooting oneself in the foot, in cases where the system actively instructs the user to check for updates to FreeBSD.

These three frames are from a screen recording of a clean test system that originated with non-patched FreeBSD 12.1-RELEASE:

2021-02-18 00:43:53 frame

2021-02-18 00:44:18 frame

2021-02-18 00:45:06 frame

there's the instruction – run freebsd-udpate fetch.

In this chroot context, I did know to ignore the instruction. I chose to run the command – in a disposable VirtualBox session – solely to capture a little of the wrongness. (I did not install what was fetched.)

Regards

Feb 18 '21 02:02 grahamperrin

… probably stated in the FreeBSD Handbook) …

– unless I'm missing something, it's not.

https://docs.freebsd.org/en/books/handbook/cutting-edge/#freebsdupdate-upgrade

Then I probably seen it on FreeBSD Forums as a general advice ...

Feb 18 '21 09:02 vermaden

System Update utility

See also

FreeBSD upgraded, no /dev/null after chroot

freebsd-update(8) without an echo of "You must be root to run this."

/dev

Old files

/dev

Old files

FreeBSD upgraded, no `/dev/null` after `chroot`

`/dev`

`/dev`