Directory and file ownership nobody:nogroup - after docker copy into sysbox enabled container

[matthewparkinsondes]

Am seeing an issue with directory and file ownership after using docker cp to copy a directory inside a sysbox enabled container.

---

Using the latest version of all system components.

os=Ubuntu 22.04.3 LTS, kernel=6.2.0-37-generic, docker=24.0.7, sysbox=0.6.2 EE (Tue Nov 14 23:13:01 UTC 2023) os=Ubuntu 22.04.3 LTS, kernel=6.2.0-37-generic, docker=24.0.7, sysbox=0.6.2 CE (Mon Jun 12 03:49:19 UTC 2023)

HWE kernel installed using the following package.

linux-image-generic-hwe-22.04/jammy-updates

The shiftfs module looks to be available.

lsmod | grep shiftfs
shiftfs                36864  0

---

Start test container.

docker run --runtime sysbox-runc -it --name z1 ubuntu:22.04 bash

---

Run following commands outside test container.

mkdir z2
echo "hi there" > z2/hello
docker cp z2 z1:/
docker exec z1 ls -l /z2

This produces the following. -rw-r--r-- 1 nobody nogroup 9 Nov 30 07:15 hello

---

We were expecting the following. -rw-r--r-- 1 root root 9 Nov 30 07:15 hello

Attempting to adjust directory and file ownership after the fact. docker exec -it z1 chown -R root:root /z2

Produces the following errors.

chown: changing ownership of '/z2/hello': Operation not permitted
chown: changing ownership of '/z2': Operation not permitted

[matthewparkinsondes]

The sysbox-docker-cp tool details this same issue for hosts where shiftfs is not present and kernel < 5.19.

https://github.com/nestybox/sysbox/blob/779889f7fa617d876bff435e15478615a69a8d32/tests/scr/sysbox-docker-cp#L26-L44

[matthewparkinsondes]

Also note this works as expected in sysbox=0.5.2 EE (Thu May 19 23:15:03 UTC 2022).

[ctalledo]

Hi Matthew (@matthewparkinsondes(, thanks for creating the issue.

I believe what's happening is that sysbox-docker-cp works when Sysbox uses shiftfs on the container's root filesystem, but does NOT work when Sysbox using ID-mapped-mounts (the shiftfs replacement). That explains why this fails in Sysbox v0.6.2 but not in Sysbox v0.5.2.

You can keep me honest by posting the findmnt and checking that the container's root filesystem (/) is not using shiftfs.

We don't have a proper solution yet, but you can work-around it by telling Sysbox to not use ID-mapped-mounts on the container rootfs (which will cause Sysbox to use shiftfs instead). That's done by passing the --disable-ovfs-on-idmapped-mount flag to sysbox-mgr.

Let me know if that helps please.

[matthewparkinsondes]

Hi Cesar (@ctalledo), thanks for the feedback.

I've posted the output below from running findmnt within the sysbox enabled container.

findmnt -n -o FSTYPE,SOURCE /
xfs    /dev/mapper/docker-8:1-774241-a7a17afa028283f51bbcc2783cefa3b9d3ca5a12aac91c0d19c4024a430f2ce6[/rootfs]

Passing the --disable-ovfs-on-idmapped-mount flag to sysbox-mgr continues to produce the same results as originally reported.

level=info msg="Starting ..."
level=info msg="Sysbox data root: /var/lib/sysbox"
level=warning msg="No kernel-headers found in host filesystem at /usr/src/linux-headers-6.2.0-37-generic. No headers will be mounted inside any of the containers."
level=info msg="Shiftfs module found in kernel: yes"
level=info msg="Shiftfs works properly: yes"
level=info msg="Shiftfs-on-overlayfs works properly: yes"
level=info msg="ID-mapped mounts supported by kernel: yes"
level=info msg="Use of overlayfs on ID-mapped mounts disabled."
level=info msg="Disallowing trusted.overlay.opaque inside container."
level=info msg="Operating in system container mode."
level=info msg="Inner container image preloading enabled."
level=info msg="Listening on /run/sysbox/sysmgr.sock"
level=info msg="Ready ..."

Also, I'd need some more info around the specific process for checking that the container's root filesystem (/) is not using shiftfs.

[ctalledo]

Hi @matthewparkinsondes,

findmnt -n -o FSTYPE,SOURCE / xfs /dev/mapper/docker-8:1-774241-a7a17afa028283f51bbcc2783cefa3b9d3ca5a12aac91c0d19c4024a430f2ce6[/rootfs]

I see the issue; your container root filesystem is on XFS, which supports ID-mapped-mounts, so Sysbox must be ID-mapping the container's rootfs .

Also, I'd need some more info around the specific process for checking that the container's root filesystem (/) is not using shiftfs.

Had it used shiftfs, then you would see a shiftfs string in the findmnt output for / above.

Passing the --disable-ovfs-on-idmapped-mount flag to sysbox-mgr continues to produce the same results as originally reported.

That makes sense, that flag only applies when the container's rootfs uses overlayfs. Unfortunately we don't have a flag that would disable ID-mapping on XFS at this time (and implementing one may not be possible, as it would require Sysbox to chown the container's rootfs to match the unprivileged uid:gid assigned to the container, and that chown can take a lot of time depending on the size of the rootfs; for overlayfs we can disable ID-mapping (though not ideal) and do that chown very fast using an overlayfs flag called metacopy-on).

But the real solution is to fix docker cp to take into account user-namespaces.

[matthewparkinsondes]

Thanks for the analysis Cesar (@ctalledo).

In the meantime we've adjusted our docker cp statements as follows.

---

docker cp config.txt ${CONTAINER}:/config/config.txt

becomes

docker exec -i ${CONTAINER} bash -c 'cat > /config/config.txt' < config.txt

---

docker cp config ${CONTAINER}:/config

becomes

tar cf - config | docker exec -i ${CONTAINER} bash -c 'cat | tar x -C /'

---

[ctalledo]

HI @matthewparkinsondes,

In the meantime we've adjusted our docker cp statements as follows.

docker cp config.txt ${CONTAINER}:/config/config.txt

becomes

docker exec -i ${CONTAINER} bash -c 'cat > /config/config.txt' < config.txt

docker cp config ${CONTAINER}:/config

becomes

tar cf - config | docker exec -i ${CONTAINER} bash -c 'cat | tar x -C /'

Wow, very clever!

FYI, there's this script called sysbox-docker-cp that we use in the Sysbox test suite, which tries to accomplish the same purpose but not as cleverly as you did. I am thinking we could leverage your technique in that script too.

[matthewparkinsondes]

Hi @ctalledo,

I am thinking we could leverage your technique in that script too.

Sounds good.

---

I recently noticed that one of our 'docker cp' scenarios operates on a partially started container launched from docker compose.

docker compose -f docker-compose.yml up --no-start

The workaround we have employed in the interim is to disable Sysbox use for this particular set of containers (luckily this was possible in the short-term for this particular use case).

This relates to the 'docker cp' workaround running 'docker exec' commands (which only works on Sysbox containers that have been fully started).

Another suitable alternative in this particular scenario may be to fully start the container, perform the 'docker exec' commands (to achieve the copy) and then restart the container to make use of the newly transferred files during container startup.

Hopefully these techniques will be of assistance to others in the interim.