operation not permitted in unpriv.link when creating an image

[taylorsilva]

OS: Ubuntu 18.04 Kernel: 4.15.0-72-generic umoci version: 0.4.5

Seems related to #222. To reproduce:

$ skopeo copy docker://opensuse/tumbleweed:latest oci:tumbleweed:latest
[output removed]
$ sudo umoci unpack --image tumbleweed:latest bundle
create runtime bundle: unpack rootfs: unpack layer: unpack entry: bin/pkill: link: unpriv.link: unpriv.wrap: linkname: link bundle/rootfs/bin/pgrep bundle/rootfs/bin/pkill: operation not permitted

Using --rootless returns the same error though.

Let me know if there's anything else you'd like me to try or help out with. I'm just playing around with skopeo, umoci, and runc right now, still learning lots.

[cyphar]

It's a bit odd that you get an error that indicates the unprivileged wrapper code was used (when you were running it as root without --rootless). But assuming you just got the error messages mixed up when pasting them, it looks like linkat(2) is giving -EPERM? There are only a few times when you might get -EPERM from linkat(2):

• You've violated the "safe hardlink" rules. This doesn't apply here, because the binary you're linking to is just a normal executable and it's all being done as the same user which created the binary.

• The file is "unlinkable" for a variety of reasons (it's append-only, immutable, a directory, the filesystem doesn't support hardlinks, and so on). None of these should apply either.

• The LSM blocked the operation. This would be very strange (given that we just created the file we're hard-linking to) but it could be possible, I guess.

Are you running this inside a container, or something like that? Is it a stock Ubuntu install? Do you have any strange AppArmor profiles?

[taylorsilva]

Thanks for the detailed response :)

I'm using an Ubuntu bionic VM brought up using vagrant on macOS, it's my personal machine. It's a stock install with various dev tools installed on it. Nothing like AppArmor is on the vm.

Here's the raw output of what I ran.

vagrant@ubuntu-bionic:~/workspace$ type skopeo
skopeo is hashed (/usr/local/bin/skopeo)

vagrant@ubuntu-bionic:~/workspace$ type umoci
umoci is hashed (/usr/local/bin/umoci)

vagrant@ubuntu-bionic:~/workspace$ skopeo copy docker://opensuse/tumbleweed:latest oci:tumbleweed:latest
Getting image source signatures
Copying blob b3d29a92b70f done
Copying config 95f5e4fd3d done
Writing manifest to image destination
Storing signatures

vagrant@ubuntu-bionic:~/workspace$ sudo umoci unpack --image tumbleweed:latest bundle
   • umoci encountered a permission error: maybe --rootless will help?
   ⨯ create runtime bundle: unpack rootfs: unpack layer: unpack entry: bin/pkill: link: link bundle/rootfs/bin/pgrep bundle/rootfs/bin/pkill: operation not permitted

Here's the umoci command again with log level set to debug

vagrant@ubuntu-bionic:~/workspace$ sudo umoci --log debug unpack --image tumbleweed:latest bundle
   • parsed mappings           map.gid=[] map.uid=[]
   • -> ws.recurse             digest=sha256:a67c2e4a8da63be605715098a7961f83749ec79b2d660be62edd54c3b036289a
   • <- ws.recurse             digest=sha256:a67c2e4a8da63be605715098a7961f83749ec79b2d660be62edd54c3b036289a
   • casext.ResolveReference(latest) got these descriptors refs=[{[{application/vnd.oci.image.manifest.v1+json sha256:a67c2e4a8da63be605715098a7961f83749ec79b2d660be62edd54c3b036289a 350 [] map[org.opencontainers.image.ref.name:latest] <nil>}]}]
   • umoci: unpacking OCI image bundle=bundle ref=latest rootfs=rootfs
   • unpacking bundle ...
   • unpack rootfs: bundle/rootfs
   • unpack layer: sha256:b3d29a92b70ffbff297cfd948c7decb3afb99ce11414e44305c89dd929579805
   • unpacking entry           path=. root=bundle/rootfs type=53
   • unpacking entry           path=bin root=bundle/rootfs type=53
   • unpacking entry           path=bin/arch root=bundle/rootfs type=50
   • unpacking entry           path=bin/awk root=bundle/rootfs type=50
   • unpacking entry           path=bin/basename root=bundle/rootfs type=50
   • unpacking entry           path=bin/bash root=bundle/rootfs type=50
   • unpacking entry           path=bin/cat root=bundle/rootfs type=50
   • unpacking entry           path=bin/chgrp root=bundle/rootfs type=50
   • unpacking entry           path=bin/chmod root=bundle/rootfs type=50
   • unpacking entry           path=bin/chown root=bundle/rootfs type=50
   • unpacking entry           path=bin/cp root=bundle/rootfs type=50
   • unpacking entry           path=bin/cpio root=bundle/rootfs type=50
   • unpacking entry           path=bin/date root=bundle/rootfs type=50
   • unpacking entry           path=bin/dd root=bundle/rootfs type=50
   • unpacking entry           path=bin/df root=bundle/rootfs type=50
   • unpacking entry           path=bin/echo root=bundle/rootfs type=50
   • unpacking entry           path=bin/egrep root=bundle/rootfs type=50
   • unpacking entry           path=bin/false root=bundle/rootfs type=50
   • unpacking entry           path=bin/fgrep root=bundle/rootfs type=50
   • unpacking entry           path=bin/fillup root=bundle/rootfs type=50
   • unpacking entry           path=bin/find root=bundle/rootfs type=50
   • unpacking entry           path=bin/gawk root=bundle/rootfs type=50
   • unpacking entry           path=bin/grep root=bundle/rootfs type=50
   • unpacking entry           path=bin/gunzip root=bundle/rootfs type=50
   • unpacking entry           path=bin/gzip root=bundle/rootfs type=50
   • unpacking entry           path=bin/ln root=bundle/rootfs type=50
   • unpacking entry           path=bin/ls root=bundle/rootfs type=50
   • unpacking entry           path=bin/md5sum root=bundle/rootfs type=50
   • unpacking entry           path=bin/mkdir root=bundle/rootfs type=50
   • unpacking entry           path=bin/mknod root=bundle/rootfs type=50
   • unpacking entry           path=bin/mktemp root=bundle/rootfs type=50
   • unpacking entry           path=bin/mv root=bundle/rootfs type=50
   • unpacking entry           path=bin/pgrep root=bundle/rootfs type=48
   • unpacking entry           path=bin/pkill root=bundle/rootfs type=49
   • umoci encountered a permission error: maybe --rootless will help?
   ⨯ create runtime bundle: unpack rootfs: unpack layer: unpack entry: bin/pkill: link: link bundle/rootfs/bin/pgrep bundle/rootfs/bin/pkill: operation not permitted

type=49 means the file is a hardlink (ASCII character 49 (took me too long to figure this out 😅 )). You figured this out already but I had fun figuring out this info 😄

I see that there are tests to ensure hardlinks are correctly unpacked. I'm not sure what's up with this particular hardlink.

[cyphar]

Alright, I'll try this in a Bionic VM.

type=49 means the file is a hardlink (ASCII character 49 (took me too long to figure this out :sweat_smile: )).

We could probably make the debugging output tell you what the type is if it's a known type.