mkosi Package manager tree ignored by `Package=` and `BuildPackage=` directive in upper layer when using `BaseTree=` and `Overlay=true`

mkosi commit the issue has been seen with

87c900f0751ccdbde83332f0f7afcf506d05cb51

Used host distribution

Debian bookworm

Used target distribution

Debian bookworm

Linux kernel version used

6.1.0-18-amd64

CPU architectures issue was seen on

x86_64

Unexpected behaviour you saw

Given the below config the build script should have an upstream nodejs installation (at this point version 18.20.3, but gets the Debian bookworm version 18.19.0. The specific versions are immaterial to the issue, but the problem is that Package= and BuildPackage= grab the wrong version because they seemingly don't get the package manager tree, while installing the same packages in a prepare script in either the final or build stage installs the correct package.

Used mkosi config

# mkosi.conf
[Distribution]
Distribution=debian

[Host]
ToolsTreeDistribution=debian
ToolsTree=default

[Output]
@Format=directory
CacheDirectory=mkosi.cache
# these three directives are here to redirect all accesses from HOME, since that is inaccessible 
# during the build
WorkspaceDirectory=../mkosi-workspace
CacheDirectory=mkosi.cache
PackageCacheDirectory=pkgcache

# mkosi.images/minimal-base/mkosi.conf
# just a perfunctory base layer
[Content]
Bootable=no
Packages=
    bash
    coreutils
    grep
    util-linux
    less

# mkosi.images/reproducer-overlay/mkosi.conf
[Config]
Dependencies=minimal-base

# [Output]
# Overlay=yes

[Distribution]
PackageManagerTrees=pkgmngr

[Content]
WithNetwork=true
BaseTrees=%O/minimal-base
Packages=
    nodejs

# mkosi.images/reproducer-overlay/pkgmngr/etc/apt/sources.list.d/nodejs.list
deb [arch=amd64 signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_18.x nodistro main

# mkosi.images/reproducer-overlay/pkgmngr/etc/apt/preferences.d/nodejs
Package: nodejs
Pin: release o=Node Source, l=Node Source
Pin-Priority: 600

# mkosi.images/reproducer-overlay/pkgmngr/etc/apt/keyrings/nodesource.gpg
# https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key

# mkosi.images/reproducer-overlay/mkosi.prepare
#!/bin/bash
# can be used to check the prepare phase
set -ex

if [[ "$1" == final ]]
then
    :
    # apt-get update
    # apt-get install nodejs
elif [[ "$1" == build ]]
then
    :
    # apt-get update
    # apt-get install nodejs
fi

# mkosi.images/reproducer-overlay/mkosi.build.chroot
#!/bin/bash
# show version and crash the build for easy reruns
set -ex

# should show v18.20.3
node --version 
exit 23

mkosi output

‣ Including configuration file /local_scratch/username/qux/reproducer/mkosi.conf
‣ Including configuration file /local_scratch/username/qux/reproducer/mkosi.images/minimal-base/mkosi.conf
‣ Including configuration file /local_scratch/username/qux/reproducer/mkosi.images/reproducer-overlay/mkosi.conf
‣ Output path minimal-base exists already. (Use --force to rebuild.)
‣ Including configuration file /tmp/tmp_6ifw55i/resources/mkosi-tools/mkosi.conf
‣ Including configuration file /tmp/tmp_6ifw55i/resources/mkosi-tools/mkosi.conf.d/10-debian-ubuntu.conf
‣ Including configuration file /tmp/tmp_6ifw55i/resources/mkosi-tools/mkosi.conf
‣ Including configuration file /tmp/tmp_6ifw55i/resources/mkosi-tools/mkosi.conf.d/10-debian-ubuntu.conf
‣ Acquiring lock on /tmp/mkosi-uidmap-lock-vq02j1_l
‣ Acquired lock on /tmp/mkosi-uidmap-lock-vq02j1_l
‣ + flock --exclusive --close /tmp/mkosi-uidmap-lock-vq02j1_l newuidmap 1299316 0 1000065537 65436 65436 141887 1 65437 1000130974 99
‣ + flock --exclusive --close /tmp/mkosi-uidmap-lock-vq02j1_l newgidmap 1299316 0 1000065537 65436 65436 9476 1 65437 1000130974 99
‣ + stat --file-system --format %T /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-_8dmauzo
‣ Copying in package manager file trees…
‣ + cp --version
‣ + cp --recursive --no-dereference --preserve=mode,links --reflink=auto --copy-contents /local_scratch/username/qux/reproducer/mkosi.images/reproducer-overlay/pkgmngr /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-_8dmauzo/pkgmngr --no-target-directory
‣ Syncing package manager metadata for reproducer-overlay image
‣ Acquiring lock on /local_scratch/username/qux/reproducer/pkgcache/cache/apt
‣ Acquired lock on /local_scratch/username/qux/reproducer/pkgcache/cache/apt
‣ Acquiring lock on /local_scratch/username/qux/reproducer/pkgcache/lib/apt
‣ Acquired lock on /local_scratch/username/qux/reproducer/pkgcache/lib/apt
‣ + apt-get -o APT::Architecture=amd64 -o APT::Architectures=amd64 -o APT::Install-Recommends=false -o APT::Immediate-Configure=off -o APT::Get::Assume-Yes=true -o APT::Get::AutomaticRemove=true -o APT::Get::Allow-Change-Held-Packages=true -o APT::Get::Allow-Remove-Essential=true -o APT::Sandbox::User=root -o Acquire::AllowReleaseInfoChange=true -o Dir::Cache=/var/cache/apt -o Dir::State=/var/lib/apt -o Dir::Log=/var/log/apt -o Dir::State::Status=/buildroot/var/lib/dpkg/status -o Dir::Bin::DPkg=/usr/bin/dpkg -o Debug::NoLocking=true -o DPkg::Options::=--root=/buildroot -o DPkg::Options::=--force-unsafe-io -o DPkg::Options::=--force-architecture -o DPkg::Options::=--force-depends -o DPkg::Options::=--no-debsig -o DPkg::Use-Pty=false -o DPkg::Install::Recursive::Minimum=1000 -o pkgCacheGen::ForceEssential=, update
Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
Hit:2 http://deb.debian.org/debian bookworm InRelease
Hit:3 http://deb.debian.org/debian-debug bookworm-debug InRelease
Hit:4 http://deb.debian.org/debian bookworm-updates InRelease
Hit:5 https://deb.nodesource.com/node_18.x nodistro InRelease
Reading package lists...
W: Skipping acquire of configured file 'main/source/Sources' as repository 'https://deb.nodesource.com/node_18.x nodistro InRelease' does not seem to provide it (sources.list entry misspelt?)
‣ + rm -rf -- /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-_8dmauzo
‣ Acquiring lock on /tmp/mkosi-uidmap-lock-ws9vio1o
‣ Acquired lock on /tmp/mkosi-uidmap-lock-ws9vio1o
‣ + flock --exclusive --close /tmp/mkosi-uidmap-lock-ws9vio1o newuidmap 1299554 0 1000065537 65436 65436 141887 1 65437 1000130974 99
‣ + flock --exclusive --close /tmp/mkosi-uidmap-lock-ws9vio1o newgidmap 1299554 0 1000065537 65436 65436 9476 1 65437 1000130974 99
‣ + mount --rbind /etc /etc --options ro
‣ + mount --rbind /opt /opt --options ro
‣ + mount --rbind /boot /boot --options ro
‣ + mount --rbind /efi /efi --options ro
‣ + mount --rbind /media /media --options ro
‣ + mount --rbind /usr /usr --options ro
‣ Building reproducer-overlay image
‣ + stat --file-system --format %T /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5
‣  Copying in package manager file trees…
‣ + cp --version
‣ + cp --recursive --no-dereference --preserve=mode,links --reflink=auto --copy-contents /local_scratch/username/qux/reproducer/mkosi.images/reproducer-overlay/pkgmngr /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/pkgmngr --no-target-directory
‣  Copying in base trees…
‣ + cp --version
‣ + cp --recursive --no-dereference --preserve=mode,links,timestamps,ownership,xattr --reflink=auto --copy-contents /local_scratch/username/qux/reproducer/mkosi.output/minimal-base /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/root --no-target-directory
‣ Found repository metadata in /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/root/var, not copying repository metadata
‣  Installing extra packages for Debian
‣ + apt-get -o APT::Architecture=amd64 -o APT::Architectures=amd64 -o APT::Install-Recommends=false -o APT::Immediate-Configure=off -o APT::Get::Assume-Yes=true -o APT::Get::AutomaticRemove=true -o APT::Get::Allow-Change-Held-Packages=true -o APT::Get::Allow-Remove-Essential=true -o APT::Sandbox::User=root -o Acquire::AllowReleaseInfoChange=true -o Dir::Cache=/var/cache/apt -o Dir::State=/var/lib/apt -o Dir::Log=/var/log/apt -o Dir::State::Status=/buildroot/var/lib/dpkg/status -o Dir::Bin::DPkg=/usr/bin/dpkg -o Debug::NoLocking=true -o DPkg::Options::=--root=/buildroot -o DPkg::Options::=--force-unsafe-io -o DPkg::Options::=--force-architecture -o DPkg::Options::=--force-depends -o DPkg::Options::=--no-debsig -o DPkg::Use-Pty=false -o DPkg::Install::Recursive::Minimum=1000 -o pkgCacheGen::ForceEssential=, install nodejs
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libbrotli1 libc-ares2 libicu72 libnghttp2-14 libnode108 libssl3 libstdc++6
  libuv1 node-acorn node-busboy node-cjs-module-lexer node-undici node-xtend
Suggested packages:
  npm
Recommended packages:
  ca-certificates nodejs-doc
The following NEW packages will be installed:
  libbrotli1 libc-ares2 libicu72 libnghttp2-14 libnode108 libssl3 libstdc++6
  libuv1 node-acorn node-busboy node-cjs-module-lexer node-undici node-xtend
  nodejs
0 upgraded, 14 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/23.9 MB of archives.
After this operation, 101 MB of additional disk space will be used.
Selecting previously unselected package libbrotli1:amd64.
(Reading database ... 3764 files and directories currently installed.)
Preparing to unpack .../libbrotli1_1.0.9-2+b6_amd64.deb ...
Unpacking libbrotli1:amd64 (1.0.9-2+b6) ...
Selecting previously unselected package libc-ares2:amd64.
Preparing to unpack .../libc-ares2_1.18.1-3_amd64.deb ...
Unpacking libc-ares2:amd64 (1.18.1-3) ...
Selecting previously unselected package libstdc++6:amd64.
Preparing to unpack .../libstdc++6_12.2.0-14_amd64.deb ...
Unpacking libstdc++6:amd64 (12.2.0-14) ...
Selecting previously unselected package libicu72:amd64.
Preparing to unpack .../libicu72_72.1-3_amd64.deb ...
Unpacking libicu72:amd64 (72.1-3) ...
Selecting previously unselected package libnghttp2-14:amd64.
Preparing to unpack .../libnghttp2-14_1.52.0-1+deb12u1_amd64.deb ...
Unpacking libnghttp2-14:amd64 (1.52.0-1+deb12u1) ...
Selecting previously unselected package libssl3:amd64.
Preparing to unpack .../libssl3_3.0.11-1~deb12u2_amd64.deb ...
Unpacking libssl3:amd64 (3.0.11-1~deb12u2) ...
Selecting previously unselected package libuv1:amd64.
Preparing to unpack .../libuv1_1.44.2-1+deb12u1_amd64.deb ...
Unpacking libuv1:amd64 (1.44.2-1+deb12u1) ...
Selecting previously unselected package node-xtend.
Preparing to unpack .../node-xtend_4.0.2-3_all.deb ...
Unpacking node-xtend (4.0.2-3) ...
Selecting previously unselected package nodejs.
Preparing to unpack .../nodejs_18.19.0+dfsg-6~deb12u1_amd64.deb ...
Unpacking nodejs (18.19.0+dfsg-6~deb12u1) ...
Selecting previously unselected package node-acorn.
Preparing to unpack .../node-acorn_8.8.1+ds+~cs25.17.7-2_all.deb ...
Unpacking node-acorn (8.8.1+ds+~cs25.17.7-2) ...
Selecting previously unselected package node-cjs-module-lexer.
Preparing to unpack .../node-cjs-module-lexer_1.2.2+dfsg-5_all.deb ...
Unpacking node-cjs-module-lexer (1.2.2+dfsg-5) ...
Selecting previously unselected package node-busboy.
Preparing to unpack .../node-busboy_1.6.0+~cs2.6.0-2_all.deb ...
Unpacking node-busboy (1.6.0+~cs2.6.0-2) ...
Selecting previously unselected package node-undici.
Preparing to unpack .../node-undici_5.15.0+dfsg1+~cs20.10.9.3-1+deb12u3_all.deb ...
Unpacking node-undici (5.15.0+dfsg1+~cs20.10.9.3-1+deb12u3) ...
Selecting previously unselected package libnode108:amd64.
Preparing to unpack .../libnode108_18.19.0+dfsg-6~deb12u1_amd64.deb ...
Unpacking libnode108:amd64 (18.19.0+dfsg-6~deb12u1) ...
Setting up libbrotli1:amd64 (1.0.9-2+b6) ...
Setting up libssl3:amd64 (3.0.11-1~deb12u2) ...
Setting up libnghttp2-14:amd64 (1.52.0-1+deb12u1) ...
Setting up node-cjs-module-lexer (1.2.2+dfsg-5) ...
Setting up libc-ares2:amd64 (1.18.1-3) ...
Setting up libuv1:amd64 (1.44.2-1+deb12u1) ...
Setting up node-xtend (4.0.2-3) ...
Setting up libstdc++6:amd64 (12.2.0-14) ...
Setting up node-busboy (1.6.0+~cs2.6.0-2) ...
Setting up libicu72:amd64 (72.1-3) ...
Setting up node-undici (5.15.0+dfsg1+~cs20.10.9.3-1+deb12u3) ...
Setting up node-acorn (8.8.1+ds+~cs25.17.7-2) ...
Setting up libnode108:amd64 (18.19.0+dfsg-6~deb12u1) ...
Setting up nodejs (18.19.0+dfsg-6~deb12u1) ...
update-alternatives: using /usr/bin/nodejs to provide /usr/bin/js (js) in auto mode
Processing triggers for libc-bin (2.36-9+deb12u7) ...
‣  Running prepare script /local_scratch/username/qux/reproducer/mkosi.images/reproducer-overlay/mkosi.prepare…
‣ + /work/prepare final
+ [[ final == final ]]
+ :
‣ + mount --no-mtab overlay /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/root --types overlay --options lowerdir=/local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/root,upperdir=/local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/build-overlay,workdir=/local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/build-overlay-workdir9t09aftz,index=off,metacopy=off,userxattr
‣  Running prepare script /local_scratch/username/qux/reproducer/mkosi.images/reproducer-overlay/mkosi.prepare in build overlay…
‣ + /work/prepare build
+ [[ build == final ]]
+ [[ build == build ]]
+ :
‣ + umount --no-mtab /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/root
‣ + mount --no-mtab overlay /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/root --types overlay --options lowerdir=/local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/build-overlay:/local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/root,upperdir=/local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/tmp/volatile-overlay13_0naug,workdir=/local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/tmp/volatile-overlay13_0naug-workdir6cckwzio,index=off,metacopy=off,userxattr
‣  Running build script /local_scratch/username/qux/reproducer/mkosi.images/reproducer-overlay/mkosi.build.chroot…
‣ + /work/build-script
+ node --version
v18.19.0
+ exit 23
‣ "bwrap --die-with-parent --proc /proc --setenv SYSTEMD_OFFLINE 1 --unsetenv TMPDIR --tmpfs /tmp --unshare-ipc --dev /dev --symlink usr/bin /bin --symlink usr/sbin /sbin --symlink usr/lib /lib --symlink usr/lib64 /lib64 --setenv PATH /scripts:/usr/bin:/usr/sbin --uid 0 --gid 0 --cap-add ALL --dir /work/src --chdir /work/src --bind /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/root /buildroot --bind /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/pkgmngr/etc /etc --ro-bind /local_scratch/username/qux/reproducer/mkosi.output/debian-tools/etc/alternatives /etc/alternatives --ro-bind /local_scratch/username/qux/reproducer/mkosi.output/debian-tools/etc/ca-certificates /etc/ca-certificates --ro-bind /local_scratch/username/qux/reproducer/mkosi.output/debian-tools/etc/pacman.d/gnupg /etc/pacman.d/gnupg --ro-bind /local_scratch/username/qux/reproducer/mkosi.output/debian-tools/etc/pki /etc/pki --bind /etc/resolv.conf /etc/resolv.conf --ro-bind /local_scratch/username/qux/reproducer/mkosi.output/debian-tools/etc/ssl /etc/ssl --ro-bind /etc/ssl/certs/ca-certificates.crt /proxy.cacert --ro-bind /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/tmp/mkosi-scripts-klz2sesx /scripts --ro-bind /local_scratch/username/qux/reproducer/mkosi.output/debian-tools/usr /usr --bind /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/root/var/cache/apt /var/cache/apt --bind /local_scratch/username/qux/reproducer/pkgcache/cache/apt/archives /var/cache/apt/archives --bind /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/root/var/lib/apt /var/lib/apt --bind /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/pkgmngr/var/log /var/log --bind /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/tmp/mkosi-var-tmp-0817a5deebae4aaa /var/tmp --bind /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/artifacts /work/artifacts --ro-bind /local_scratch/username/qux/reproducer/mkosi.images/reproducer-overlay/mkosi.build.chroot /work/build-script --ro-bind /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/tmp/tmp4wqxn0q5 /work/config.json --bind /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/dest /work/dest --bind /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/staging /work/out --bind /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/packages /work/packages --bind /local_scratch/username/qux/reproducer /work/src sh -c 'chmod 1777 /dev/shm && chmod 755 /etc && exec $0 "$@"' bwrap --dev-bind / / --tmpfs /buildroot/run --tmpfs /buildroot/tmp --bind /var/tmp /buildroot/var/tmp --proc /buildroot/proc --dev /buildroot/dev --ro-bind-try /buildroot/etc/machine-id /buildroot/etc/machine-id --dir /buildroot/run/user/0 --ro-bind-try /buildroot/etc/passwd /etc/passwd --ro-bind-try /buildroot/etc/group /etc/group --ro-bind-try /buildroot/etc/shadow /etc/shadow --ro-bind-try /buildroot/etc/gshadow /etc/gshadow sh -c 'chmod 1777 /buildroot/tmp /buildroot/var/tmp /buildroot/dev/shm && chmod 755 /buildroot/run && mkdir -m 755 /buildroot/run/host && echo mkosi >/buildroot/run/host/container-manager && exec $0 "$@"' sh -c 'trap '"'"'rm -rf /buildroot/work'"'"' EXIT && mkdir -p -m 755 /buildroot/work /buildroot/run/systemd /buildroot/run/systemd/resolve && $0 "$@"' bwrap --dev-bind /buildroot / --setenv container mkosi --setenv HOME / --setenv PATH /work/scripts:/usr/bin:/usr/sbin --ro-bind-try /etc/resolv.conf /etc/resolv.conf --bind /work /work --chdir /work/src --setenv BUILDROOT / sh -c '[ "$0" = "sh" ] && [ $# -eq 0 ] && exec bash -i || exec $0 "$@"' /work/build-script" returned non-zero exit code 23.
‣ + umount --no-mtab /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5/root
‣ + rm -rf -- /local_scratch/username/qux/mkosi-workspace/mkosi-workspace-dzoomip5
‣  (Fixing ownership of package manager cache directory)
‣ + chown --recursive 65436:65436 /local_scratch/username/qux/reproducer/pkgcache/cache/apt
‣ + chown --recursive 65436:65436 /local_scratch/username/qux/reproducer/pkgcache/lib/apt
Traceback (most recent call last):
  File "/local_scratch/username/qux/reproducer/venv/lib/python3.11/site-packages/mkosi/run.py", line 60, in uncaught_exception_handler
    yield
  File "/local_scratch/username/qux/reproducer/venv/lib/python3.11/site-packages/mkosi/run.py", line 101, in fork_and_wait
    target(*args, **kwargs)
  File "/local_scratch/username/qux/reproducer/venv/lib/python3.11/site-packages/mkosi/__init__.py", line 4493, in run_build
    build_image(Context(args, config, workspace=workspace, resources=resources))
  File "/local_scratch/username/qux/reproducer/venv/lib/python3.11/site-packages/mkosi/__init__.py", line 3696, in build_image
    run_build_scripts(context)
  File "/local_scratch/username/qux/reproducer/venv/lib/python3.11/site-packages/mkosi/__init__.py", line 668, in run_build_scripts
    run(
  File "/local_scratch/username/qux/reproducer/venv/lib/python3.11/site-packages/mkosi/run.py", line 150, in run
    with spawn(
  File "/usr/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/local_scratch/username/qux/reproducer/venv/lib/python3.11/site-packages/mkosi/run.py", line 351, in spawn
    raise subprocess.CalledProcessError(returncode, cmdline)
subprocess.CalledProcessError: Command '['/work/build-script']' returned non-zero exit status 23.

May 27 '24 09:05 behrmann

debug.log

May 27 '24 09:05 behrmann

After updating the reproducer, it seems to work now.

Nov 22 '24 14:11 behrmann