Kaniko build: error removing lib to make way for new symlink

[mariovor]

Environment: AWS; Ubuntu 22.04 Shifts:

modinfo shiftfs
filename:       /lib/modules/5.15.0-1005-aws/updates/dkms/shiftfs.ko
license:        GPL v2
description:    id shifting filesystem
author:         Christian Brauner <[email protected]>
author:         Seth Forshee <[email protected]>
author:         James Bottomley
alias:          fs-shiftfs
srcversion:     B0C2D82DE327B38F653B659
depends:        
retpoline:      Y
name:           shiftfs
vermagic:       5.15.0-1005-aws SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         ip-172-20-4-96 Secure Boot Module Signature key
sig_key:        58:98:3B:C9:DD:E1:B9:01:AD:F4:71:01:C5:1A:F0:62:1F:DF:C6:20
sig_hashalgo:   sha512
signature:      16:43:E5:3F:EA:E3:C5:23:87:16:F4:9B:CE:9B:7A:7D:6B:45:D9:23:
		F3:45:E6:0B:19:71:E7:24:05:12:60:B2:33:01:06:51:BA:B5:81:AF:
		C1:BE:89:DB:FD:22:DD:7E:86:B1:B2:58:9F:94:F1:A9:93:76:90:4D:
		6C:9B:BB:F1:2B:BE:6D:81:CC:11:74:6B:53:57:84:44:9F:17:20:3A:
		C1:17:B8:70:BB:0D:E1:58:6B:10:1B:54:05:0C:ED:61:4F:8F:A6:9C:
		F5:B0:AA:39:95:DA:A2:B9:43:AC:17:1A:65:52:E9:92:B9:B0:6F:A2:
		E7:18:92:C1:A8:16:2A:24:B5:7A:C3:69:9B:9C:CC:23:E2:50:B7:CD:
		8A:15:FB:75:0D:90:AF:1C:28:79:B1:D9:EA:5C:AE:A6:1F:61:07:73:
		3E:4E:8E:B3:19:CD:7A:31:11:A7:32:3E:E0:80:A6:9F:72:F5:6A:5B:
		D1:E8:EA:C0:09:5A:53:E3:62:F3:D8:67:0E:33:DC:36:0E:76:E8:BB:
		21:16:CB:AA:74:C7:7B:DC:BA:F4:27:35:E7:03:EA:B1:F0:13:B1:66:
		33:00:CB:E3:50:32:E9:1F:B6:6D:92:F7:BD:4B:7E:CD:34:DB:90:65:
		12:CB:AD:AE:EE:16:E9:1B:D1:A4:91:8C:4B:74:59:E4

Sysbox:

systemctl status sysbox
● sysbox.service - Sysbox container runtime
     Loaded: loaded (/lib/systemd/system/sysbox.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-06-14 08:05:15 UTC; 6min ago
       Docs: https://github.com/nestybox/sysbox
   Main PID: 8260 (sh)
      Tasks: 2 (limit: 521)
     Memory: 444.0K
        CPU: 52ms
     CGroup: /system.slice/sysbox.service
             ├─8260 /bin/sh -c "/usr/bin/sysbox-runc --version && /usr/bin/sysbox-mgr --version && /usr/bin/sysbox-fs --version && /bin/sleep infinity"
             └─8283 /bin/sleep infinity

Jun 14 08:05:15 ip-172-20-6-156 sh[8269]:         version:         0.5.2
Jun 14 08:05:15 ip-172-20-6-156 sh[8269]:         commit:         ea1b7db91031355cb10b850125e0d6502dc38962
Jun 14 08:05:15 ip-172-20-6-156 sh[8269]:         built at:         Wed May 18 19:49:36 UTC 2022
Jun 14 08:05:15 ip-172-20-6-156 sh[8269]:         built by:         Rodny Molina
Jun 14 08:05:15 ip-172-20-6-156 sh[8274]: sysbox-fs
Jun 14 08:05:15 ip-172-20-6-156 sh[8274]:         edition:         Community Edition (CE)
Jun 14 08:05:15 ip-172-20-6-156 sh[8274]:         version:         0.5.2
Jun 14 08:05:15 ip-172-20-6-156 sh[8274]:         commit:         95a773a6ea3920f7ab454f1583465c7aea4c701f
Jun 14 08:05:15 ip-172-20-6-156 sh[8274]:         built at:         Wed May 18 19:49:30 UTC 2022
Jun 14 08:05:15 ip-172-20-6-156 sh[8274]:         built by:         Rodny Molina

Dockerfile:

FROM ubuntu:20.04
RUN apt-get update \
	&& DEBIAN_FRONTEND=noninteractive apt-get install -y \
	libxerces-c3.2 python3 curl \
	&& apt-get clean \
	&& rm -rf /var/lib/apt/lists/*

Steps to reproduce: Start container:

docker run -v $PWD:/app --rm  -it --entrypoint="" --runtime=sysbox-runc gcr.io/kaniko-project/executor:v1.8.1-debug /bin/sh

Run Kaniko

/kaniko/executor --dockerfile /app/Dockerfile --no-push

Error

/workspace # /kaniko/executor --dockerfile /app/Dockerfile --no-push
INFO[0000] Retrieving image manifest ubuntu:20.04       
INFO[0000] Retrieving image ubuntu:20.04 from registry index.docker.io 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest ubuntu:20.04       
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Unpacking rootfs as cmd RUN apt-get update 	&& DEBIAN_FRONTEND=noninteractive apt-get install -y 	libxerces-c3.2 python3 curl 	&& apt-get clean 	&& rm -rf /var/lib/apt/lists/* requires it. 
error building image: error building stage: failed to get filesystem from image: error removing lib to make way for new symlink: unlinkat //lib/modules/5.15.0-1005-aws/modules.builtin.modinfo: read-only file system

Running with default runtime works.

Let me know if you need more information.

[ctalledo]

Hi @mariovor, thanks for giving Sysbox a shot and for filing the issue.

On a quick look, it seems Kaniko (running inside the Sysbox container) is failing as it's trying to remove file lib/modules/5.15.0-1005-aws/modules.builtin.modinfo file and it's hitting an error because Sysbox implicitly mounts the host's /lib/modules/<kernel-ver> into the container as read-only (in this way it's different than other container runtimes).

Sysbox does this implicit mount because several programs that typically run inside Sysbox containers use the files under /lib/modules/<kernel-ver>.

One work-around (if you are open to it) would be to explicitly mount a dummy Docker volume over the container's /lib/modules/<kernel-ver>, as follows:

docker run -v $PWD:/app --rm  -it --entrypoint="" --runtime=sysbox-runc -v dummyvol:/lib/modules/5.15.0-1005-aws gcr.io/kaniko-project/executor:v1.8.1-debug /bin/sh

This way, inside the container the directory /lib/modules/5.15.0-1005-aws will now be read-write and empty, and Kaniko should not complain any more.

However, this will not work in Kaniko in fact expects the container's /lib/modules/5.15.0-1005-aws directory to hold the kernel module files (since we mounted a dummy volume on it). In that case, you would need to create a copy of /lib/modules/5.15.0-1005-aws into some other dir on the host, and mount that other dir into the Sysbox container. This way Kaniko will see the original contents of the /lib/modules/<kernel> dir and can modify them as needed.

I don't recommend mounting the host's /lib/modules/5.15.0-1005-aws into the container as read-write, as otherwise the container can mess up the host's config (e.g., if it decides to delete files in there, like Kaniko is apparently doing).

Hope that makes sense.

[mariovor]

Thanks @ctalledo for the analysis. We are seeing this error in our GitLab Runners which we swiched some time ago to Sysbox. I will try out your workaround, however I'am suprised that Kaniko is trying to remove anything in lib/modules/5.15.0-1005-aws. Thats sound really strange to me. Maybe that is a bug on their side.

[ctalledo]

I will try out your workaround, however I'am suprised that Kaniko is trying to remove anything in lib/modules/5.15.0-1005-aws. Thats sound really strange to me. Maybe that is a bug on their side.

I was surprised too, but that's clearly what it's doing (apparently is trying to replace the file with a symlink):

error building image: error building stage: failed to get filesystem from image: error removing lib to make way for new symlink: unlinkat //lib/modules/5.15.0-1005-aws/modules.builtin.modinfo: read-only file system

(unlinkat is Linux jargon for removing a file).

Let me know what you find out @mariovor.

Thanks!