`--copy-up` breaks relative symlinks in child directories

[terencehonles]

I was trying to use docker rootless with rootlesskit and I identified why x509 certificates were not being loaded (and therefore not allowing me to pull docker images). The rootlesskit command is run with --copy-up=/etc --copy-up=/run which on my system the x509 certificates are symlinked:

/etc/ssl/
├── ca-bundle.pem -> ../../var/lib/ca-certificates/ca-bundle.pem
├── certs -> ../../var/lib/ca-certificates/pem
├── certs.old -> ../../var/lib/ca-certificates/pem
└── ...

But those links are broken when running inside the rootless child:

/etc/
├── ...
├── .ro780432485
│   ├── ssl
│   │   ├── ca-bundle.pem -> ../../var/lib/ca-certificates/ca-bundle.pem
│   │   ├── certs -> ../../var/lib/ca-certificates/pem
│   │   ├── certs.old -> ../../var/lib/ca-certificates/pem
│   │   └── ...
├── ...
├── ssl -> .ro780432485/ssl
└── ...

I'm not sure if this is a known issue, but I didn't see it in the github issues and it took me awhile to debug the underlying issue (I lifted the docker pull code to try connecting to the docker registry until I narrowed down to the fact that the x509 certificates were not being loaded).

I'm slightly surprised to see a relative link to resolve /var/lib, but in general relative links should be supported and I assume this is not going to be a straightforward fix. If it might be I don't mind playing around with it and pushing a PR.

[AkihiroSuda]

Thanks for reporting, I guess we should have an option to bind-mount /etc/ssl from the parent

[AkihiroSuda]

Hmm, tumbleweed seems massively using relative symlinks for /etc ... 😞 cc @cyphar

# grep VERSION_ID /etc/os-release
VERSION_ID="20210222"

# find /etc -type l | xargs ls -l | grep -F ..
lrwxrwxrwx 1 root root 21 Feb 22 23:20 /etc/os-release -> ../usr/lib/os-release
lrwxrwxrwx 1 root root 43 Jan 15 19:22 /etc/ssl/ca-bundle.pem -> ../../var/lib/ca-certificates/ca-bundle.pem
lrwxrwxrwx 1 root root 33 Jan 15 19:22 /etc/ssl/certs -> ../../var/lib/ca-certificates/pem
lrwxrwxrwx 1 root root 25 Feb 11 12:27 /etc/termcap -> ../usr/share/misc/termcap

[AkihiroSuda]

Workaround

--- dockerd-rootless.sh.OLD     2021-02-26 07:46:56.137467327 +0100
+++ dockerd-rootless.sh 2021-02-26 07:51:09.832257325 +0100
@@ -95,5 +95,12 @@
        # remove the symlinks for the existing files in the parent namespace if any,
        # so that we can create our own files in our mount namespace.
        rm -f /run/docker /run/containerd /run/xtables.lock
+
+       # Workaround for https://github.com/rootless-containers/rootlesskit/issues/225
+       realpath_etc_ssl=$(realpath /etc/ssl)
+       rm -f /etc/ssl
+       mkdir /etc/ssl
+       mount --rbind ${realpath_etc_ssl} /etc/ssl
+
        exec dockerd $@
 fi

[cyphar]

I found this out recently -- all symlinks in openSUSE rpm packages apparently get converted to relative symlinks -- I have no idea why because it's done at the end of the build as part of the rpmbuild post-processing stage.

[terencehonles]

Thanks @AkihiroSuda for the suggestion. I tested it and it looks good to me, but not sure what the plans are to make sure this doesn't break other relative symlinks. Is there a reason mount -rbind isn't used instead of symlinks? I'm not sure the implications and I would imagine this might be related to performance?

Anyways glad I can now poke around at running docker rootlessly :slightly_smiling_face:

[AkihiroSuda]

Is there a reason mount -rbind isn't used instead of symlinks?

That makes /etc unwritable.

[AkihiroSuda]

Opened Moby PR https://github.com/moby/moby/pull/42457

[DavidTorresOcana]

Workaround

--- dockerd-rootless.sh.OLD     2021-02-26 07:46:56.137467327 +0100
+++ dockerd-rootless.sh 2021-02-26 07:51:09.832257325 +0100
@@ -95,5 +95,12 @@
        # remove the symlinks for the existing files in the parent namespace if any,
        # so that we can create our own files in our mount namespace.
        rm -f /run/docker /run/containerd /run/xtables.lock
+
+       # Workaround for https://github.com/rootless-containers/rootlesskit/issues/225
+       realpath_etc_ssl=$(realpath /etc/ssl)
+       rm -f /etc/ssl
+       mkdir /etc/ssl
+       mount --rbind ${realpath_etc_ssl} /etc/ssl
+
        exec dockerd $@
 fi

Nice work around. Certificates might not always be in /etc/ssl but users might setup variable REQUESTS_CA_BUNDLE to other directory.

The following cover this case too:

# Workaround for "x509: certificate signed by unknown authority" on openSUSE Tumbleweed.
# https://github.com/rootless-containers/rootlesskit/issues/225
rm -f /etc/ssl
mkdir /etc/ssl
if [[ -z ${REQUESTS_CA_BUNDLE} ]];			
then
	realpath_etc_ssl=$(realpath /etc/ssl)
	mount --rbind ${realpath_etc_ssl} /etc/ssl
else
	mkdir /etc/ssl/certs
	touch /etc/ssl/certs/ca-certificates.crt
	mount --rbind ${REQUESTS_CA_BUNDLE} /etc/ssl/certs/ca-certificates.crt
fi