localhost is resolved to `::1` (IPv6) instead of `127.0.0.1`

[GuLinux]

Description

Since upgrading to docker desktop v4.29.0, we've noticed that the host resolution for localhost is resolving to the IPv6 version (::1) instead of IPv4 (127.0.0.1).

Specifically, with a nslookup I can notice that the order of IPv4 and IPv6 resolution is inverted, and in v4.29.0 the latter is prioritised.

This is not mentioned in the changelog, so I assume it's a bug, especially since IPv6 networks are not supported on Docker for Mac. In our environment services start with IPv4 stack, so having healthcheck calls to http://localhost/status result to the containers status being "unhealthy".

Reproduce

• Install Docker Desktop v4.28.0
• Run:

docker run -it --rm alpine sh
/ # nslookup localhost
Server:		192.168.65.7
Address:	192.168.65.7:53

Non-authoritative answer:
Name:	localhost
Address: 127.0.0.1


Non-authoritative answer:
Name:	localhost
Address: ::1

/ # ping localhost
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.330 ms

• Install Docker Desktop v4.29.0
• Run:

docker run -it --rm alpine sh
/ # nslookup localhost
Server:		192.168.65.7
Address:	192.168.65.7:53

Non-authoritative answer:
Name:	localhost
Address: ::1

Non-authoritative answer:
Name:	localhost
Address: 127.0.0.1

/ # ping localhost
PING localhost (::1): 56 data bytes
64 bytes from ::1: seq=0 ttl=64 time=0.330 ms

Expected behavior

localhost should resolve to the IPv4 address (127.0.0.1)

docker version

Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:14:46 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.29.0 (145265)
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:02 2024
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    26.0.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1-desktop.1
    Path:     /Users/mgu06/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1-desktop.1
    Path:     /Users/mgu06/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.27
    Path:     /Users/mgu06/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/mgu06/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     /Users/mgu06/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     /Users/mgu06/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     /Users/mgu06/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/mgu06/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.6.3
    Path:     /Users/mgu06/.docker/cli-plugins/docker-scout

Server:
 Containers: 8
  Running: 1
  Paused: 0
  Stopped: 7
 Images: 134
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.22-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 4.071GiB
 Name: docker-desktop
 ID: a5de42d5-7433-4f06-9b3c-110fa899822b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/mgu06/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: true
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

1947FF30-F1E0-456C-A352-91A4B412F253/20240501082456

Additional Info

No response

[dgageot]

cc @djs55

[Sergio-bzh]

Hi every one, I hope you're all doing well as your beloved ones.

I'm facing the same issue as GuLinux since last update :-(

[robmry]

Thank you for the nice clear report and examples - I think I see what's happening.

To give some background ...

In earlier releases, the Docker engine would only enable IPv6 on the loopback interface when a container was connected to an IPv6-enabled network. When it was disabled, there was no ::1 address in the container. So, when a container was connected to its first IPv6-enabled network, or disconnected from the last, the ::1 address would appear or disappear.

But, the /etc/hosts file always contained an entry for ::1 localhost (and DNS would respond with ::1). That caused problems for some applications when there was no ::1 address.

That DNS response for an absent ::1 is what you're seeing with Docker Desktop v4.28.0. The nslookup command is making two requests, one for an A record and another for an AAAA record. Those requests race, and which response gets back to nslookup first is just luck. (You'll probably see the order change if you run the nslookup command a few times.)

As part of ongoing work to improve Docker's support for IPv6 - by default, we now leave IPv6 enabled on the loopback interface, so it always has a ::1 address. That makes network connect/disconnect more like connecting an ethernet cable to a physical host, the interface may or may not get an IPv6 address - but it doesn't cause reconfiguration of the host's loopback interface.

So ...

I think the change you're seeing is probably because ::1 is now present on the loopback interface.

If the ::1 address didn't exist, the healthcheck request went to 127.0.0.1 instead. (You're seeing the inverse of the problem caused by the absence of ::1.)

In which case, there are a couple of options ...

If the healthcheck only works on IPv4, it would probably be best to change the healthcheck URL to http://127.0.0.1/status.

Alternatively, you can disable IPv6 in the container so that it never has a ::1 (or an /etc/hosts entry for ::1). To do that, use --sysctl=net.ipv6.conf.all.disable_ipv6=1 in the docker run command. Or, the equivalent sysctls option in a compose file.

Hopefully that makes sense, and helps? Let me know!

[GuLinux]

Hi, thank you for your reply.

Yes, I thought of using 127.0.0.1 instead as a workaround, and yes, we could potentially alter /etc/hosts in the containers. Both of these look like workarounds though, I was hoping to have an actual solution in Docker itself, as this is clearly a breaking change (I wouldn't expect every single image in docker hub to have their respective /etc/hosts file altered because of this)

[robmry]

The /etc/hosts file is generated by the engine and mounted into the container as it starts up. It has always included an entry for ::1.

Editing the /etc/hosts file in a running container, removing IPv6 entries if not required, may be an option. But yes, it would be a fiddly workaround (so I didn't mention it before).

The change is that the loopback interface now keeps the ::1 address, even while the container is not connected to an IPv6 network. Removing ::1 from the interface, depending on which networks were connected, was problematic and unusual behaviour - very unlike a Linux host. The change to the new default behaviour is needed as we make improvements to the way IPv6 is configured.

Explicitly using the IPv4 address for an IPv4-only service will always work.

The old behaviour, no ::1, can be restored for an IPv4-only container using --sysctl=net.ipv6.conf.all.disable_ipv6=1.

[GuLinux]

Thanks, if that's the case, we'll start using 127.0.0.1 as for now it's the easiest solution for us

[Sergio-bzh]

Hi you all, Thanks for explanations and workaround.

👍

[tscully49]

Facing the same issue here. This broke some automation of ours and we lost hours of time to debugging.

[sammyhk]

Related to change of https://github.com/moby/moby/pull/47062 in Docker 26+

[sammyhk]

Thank you for the nice clear report and examples - I think I see what's happening.

To give some background ...

In earlier releases, the Docker engine would only enable IPv6 on the loopback interface when a container was connected to an IPv6-enabled network. When it was disabled, there was no ::1 address in the container. So, when a container was connected to its first IPv6-enabled network, or disconnected from the last, the ::1 address would appear or disappear.

But, the /etc/hosts file always contained an entry for ::1 localhost (and DNS would respond with ::1). That caused problems for some applications when there was no ::1 address.

That DNS response for an absent ::1 is what you're seeing with Docker Desktop v4.28.0. The nslookup command is making two requests, one for an A record and another for an AAAA record. Those requests race, and which response gets back to nslookup first is just luck. (You'll probably see the order change if you run the nslookup command a few times.)

As part of ongoing work to improve Docker's support for IPv6 - by default, we now leave IPv6 enabled on the loopback interface, so it always has a ::1 address. That makes network connect/disconnect more like connecting an ethernet cable to a physical host, the interface may or may not get an IPv6 address - but it doesn't cause reconfiguration of the host's loopback interface.

So ...

I think the change you're seeing is probably because ::1 is now present on the loopback interface.

If the ::1 address didn't exist, the healthcheck request went to 127.0.0.1 instead. (You're seeing the inverse of the problem caused by the absence of ::1.)

In which case, there are a couple of options ...

If the healthcheck only works on IPv4, it would probably be best to change the healthcheck URL to http://127.0.0.1/status.

Alternatively, you can disable IPv6 in the container so that it never has a ::1 (or an /etc/hosts entry for ::1). To do that, use --sysctl=net.ipv6.conf.all.disable_ipv6=1 in the docker run command. Or, the equivalent sysctls option in a compose file.

Hopefully that makes sense, and helps? Let me know!

I think the root cause is docker generated /etc/hosts with duplicated entry of localhost which can resolve to IPv4 127.0.0.1 or IPv6 ::1:

$ cat /etc/hosts
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
10.0.5.6        1f0663d9fd4f

While a bare metal Linux machine will not have such duplicated localhost entry, reference from Ubuntu 22.04 bare metal IPv6 enabled server:

$ cat /etc/hosts
127.0.0.1	localhost
127.0.1.1	server-hostname

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

[robmry]

Hi @sammyhk - that's interesting, but not universal ...

Debian 12.5:

127.0.0.1	localhost
127.0.1.1	debian.myguest.virtualbox.org	debian

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

CentOS Stream 9:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

openSUSE Tumbleweed:

127.0.0.1	localhost localhost.localdomain
::1		localhost localhost.localdomain ipv6-localhost ipv6-loopback

# special IPv6 addresses
fe00::0         ipv6-localnet

ff00::0         ipv6-mcastprefix
ff02::1         ipv6-allnodes
ff02::2         ipv6-allrouters
ff02::3         ipv6-allhosts

Only having 127.0.0.1 localhost like Ubuntu should usually work. But, particularly as Docker has always included ::1, removing it now might be another breaking change for some users.

[sammyhk]

@robmry Hmm, interesting. I checked in Docker 24.0.7, as before https://github.com/moby/moby/pull/47062 , the /etc/hosts still contains the ::1 localhost entry but ping localhost always return 127.0.0.1. After that change in Docker 26+, ping localhost now always return ::1.

I really don't understand why have this behavior now...

[robmry]

I really don't understand why have this behavior now...

Is it not this? ... https://github.com/docker/for-mac/issues/7269#issuecomment-2107333702

[sammyhk]

Is it not this? ... #7269 (comment)

Yes, I checked after adding --sysctl=net.ipv6.conf.all.disable_ipv6=1 can revert back to the previous behavior, with one thing changed which /etc/hosts does not contains IPv6 stuff (Docker 26- will have those IPv6 stuff even no IPv6 enabled), which is good. What I don't understand is, even when I added back those IPv6 stuff in /etc/hosts, ping localhost can always resolved to 127.0.0.1 instead of ::1. To illustrate (--sysctl=net.ipv6.conf.all.disable_ipv6=1 set):

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
 ...

# ping localhost
PING localhost (127.0.0.1): 56 data bytes
...
 
# ping -6 localhost
ping: bad address 'localhost'
 
## after this line, edited /etc/hosts to add
::1 localhost ip6-localhost ip6-loopback

# ping localhost
PING localhost (127.0.0.1): 56 data bytes
...
 
# ping -6 localhost
PING localhost (::1): 56 data bytes
ping: sendto: Address not available

When in system which --sysctl=net.ipv6.conf.all.disable_ipv6=1 not set, ping localhost always resolve to ::1, seems like IPv6 have higher preceding than IPv4.

[robmry]

seems like IPv6 have higher preceding than IPv4.

Yes, that's RFC-6724. With glibc, the precedence is configurable.

[Yaytay]

We use swarm. Looking into what it will take to set the sysctl for swarm in our compose files I find that the spec (https://github.com/docker/compose/blob/v1/docs/Compose%20file%20reference%20(legacy)/version-3.md) says this: IPv6 options do not currently work in swarm mode.

I'm finding it hard to believe that https://github.com/moby/moby/pull/47062 solves more problems than it causes :(

Don't we need a way to disable IPv6 for all containers on a host?

[robmry]

IPv6 options do not currently work in swarm mode.

The --sysctl option isn't IPv6-specific ... docker service create --sysctl net.ipv6.conf.all.disable_ipv6=1 ... works, as well as the equivalent in a compose file:

services:
  myservice:
      sysctls:
        - net.ipv6.conf.all.disable_ipv6=1