Cannot start containers after Docker engine v26.0.2

[joeabbey]

Bug description

Any attempts to start a container result in:

open /proc/sys/net/ipv6/conf/eth0/disable_ipv6: read-only file system

Steps to reproduce

Run:

docker run ubuntu

Output:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: failed to add interface vethc71242a to sandbox: error setting interface "vethc71242a" IPv6 to <nil>: failed to configure ipv6: failed to disable IPv6 forwarding for container's interface eth0: open /proc/sys/net/ipv6/conf/eth0/disable_ipv6: read-only file system: unknown.
ERRO[0005] error waiting for container: context canceled 

Workspace affected

dakotaxyz-dakota-2w017zk6yxn

Expected behavior

the container starts

Example repository

No response

Anything else?

No response

[joeabbey]

I've also filed an issue with moby since this seems to be related to https://github.com/moby/moby/security/advisories/GHSA-x84c-p2g9-rqv9

[joeabbey]

pinned our docker-ce and docker-ce-cli version to 26.0.1

[Cellobita]

Same here. I've spent several hours trying to get this working inside a systemd-nspawnd jail.

The suggested workaround, using --sysctl=net.ipv6.conf.all.disable_ipv6=1 in the docker create or docker run commands does not work, neither does adding the option to the sysctls section in the compose.yaml file.

The last docker-ce / docker-ce-cli version working for me is also 26.0.1

[robmry]

Hello all - as @joeabbey says, this is related to the moby security advisory (and I'm the guilty party) ... the change is to make sure IPv6 is disabled on an interface, when only IPv4 is configured in the docker network.

At the moment, dockerd always tries to write to the "/proc" file. We'll make it so that it only tries to write if it needs to change the current value. That might help in some cases.

Also - when IPv6 needs to be disabled but dockerd can't do it, in the short term, we can provide a dockerd command line / daemon.json flag to restore the old behaviour, so that it just logs the problem. Then, gitpod will need to set that flag, if it's the appropriate thing to do.

(Once we've made docker's IPv6 configuration simpler and more flexible, we'll remove the override and require IPv6 to be configured in the docker network when IPv6 can't be disabled on the interface.)

So, a couple of questions ...

• Does that sound like it'll solve the problem?
• What is the problem?! I'd like to understand the use-case better, why is "/proc/sys/net" file read-only for dockerd? (I guess there's some extra isolation, but how is it implemented?)

@Cellobita - with systemd-nspawnd, are you running dockerd in the jail? I guess you have a configuration that gives dockerd enough permissons to be able to create and configure networks, but blocks access to some or all of the "/proc"?

[Cellobita]

@robmry, thank you for your reply - yes, I'm running docker inside a jail; the server is running TrueNAS SCALE (Linux-based) and the jailmaker script that a TN user put together and maintains.

I did manage to get it working with systemd_nspawn_user_args=--capability=CAP_NET_ADMIN - I had to put together a quick and dirty solution, and though it works, I'm certainly not sure of the potential risks involved.

[robmry]

Ah, yes - CAP_NET_ADMIN will make the /proc files writable, amongst other things - including allowing dockerd to configure iptables and its networking, which it's always needed to do. So, I'd be interested to know how that works for older moby releases - I'll take a look at the script, thank you.

[Cellobita]

The jailmaker script does have a docker_compatible parameter while creating a jail, which in the past one was supposed to enable if installing docker inside the jail, but the parameter is deprecated and will be removed in the future. I could have used it to spin my jail up, but I'd rather avoid using deprecated stuff, obviously.

[Cellobita]

There was a discussion about this stuff, over the past few days: https://github.com/Jip-Hop/jailmaker/issues/119 - some (most?) of it is way over my head, but you'll probably find it useful, @robmry

[robmry]

Thank you!

Is gitpod using host networking - or are these slightly different problems?

(I think dockerd needs CAP_NET_ADMIN for its basic networking setup, so the gitpod problem probably isn't that it's running on a host without that capability. But, perhaps I'm missing something.)

[Cellobita]

I'm not sure. TBH, a Google search of this issue was singularly barren, yesterday (or I did not drink enough coffee during the day) - this is the only thread I could find with the exact error I was getting, my posting here was kind of a Hail Mary...

[svenefftinge]

Sorry for the delay. We are looking into this.

[robmry]

Hi @svenefftinge - to summarise from the moby side ...

The problem is with an environment where "/proc/sys/net" is read-only. It'd normally be read-write on a host running dockerd, because dockerd requires CAP_NET_ADMIN - but the jailmaker users have an environment where it is mounted read-only.

I guess it'll be similar for gitpod - it'd be good to confirm, and I'd be interested to know how it's set up.

The bad change in moby tries to disable IPv6 by writing to the "/proc/sys/net" file, even if it's already disabled. We'll stop it from doing that. If we can't disable IPv6, for an interface on an IPv4-only network - we'll add an environment variable override that tells dockerd to ignore the problem (requiring explicit action, because of the GHSA @joeabbey lined to above). Other options will be to explicitly enable IPv6 on the network, or make "/proc/sys/net" read-write.

Those changes should be available soon, in moby 26.1.1.

(Then, in a near-future moby release once we've made IPv6 easier to configure, we plan to remove support for the new env-var - and require that IPv6 is explicitly enabled on a network if dockerd can't disable it for the interface.)

[svenefftinge]

Ok, I think we will just need to make sure that we don't use 26.1 for now. Yes, we have a lot of special isolation in place. Also we are running docker in docker, is the "/proc/sys/net" not always read only in those cases?

[robmry]

docker-in-docker is normally run with --privileged, so the inner docker gets a read-write filesystem. But, it seems if the host running the outer docker has a read-only filesystem, it stays read-only on the inner one (but both dockerd's still get the other CAP_NET_ADMIN rights they need).

In current releases - if IPv6 is enabled on the host but disabled in a docker network, and dockerd can't disable it by writing the /proc file, IPv6 stays enabled on the IPv4 only interface. That's what the security advisory was about, and the reason for the change.

It's a particular issue for ipvlan/macvlan docker networks, where the interface might automatically get a SLAAC assigned IPv6 address on the host's network. I guess that won't be an issue in your case (?). So, setting the environment variable before starting docker will probably be a good option.

[kylos101]

👋 hey there, we're preparing a fix in https://github.com/gitpod-io/workspace-images/pull/1339. It'll pin docker-ce and docker-ce-cli to version 26.0.1 (ref), and is presently making it's way through test automation, before lands on main and is then published to Docker Hub.

For posterity, we publish latest images weekly on Mondays. Today's run is here, and this is when the update from 26.0.1 to 26.1.0 occurred.

[kylos101]

👋 hello again, there are a a couple workarounds that I'd like to share for the meantime:

1. Use a specific tag for your workspace image. gitpod/workspace-full:2024-04-16-12-16-24 is a great choice, because it includes docker 26.0.1.
2. Use a custom image, and install 26.0.1 (refer to this PR to see the specific versions, assuming you're using ubuntu).

[kylos101]

👋 The latest tag has been updated. If you are still having trouble using workspace images provided by Gitpod, please rebuild your workspace image and test again.