colima icon indicating copy to clipboard operation
colima copied to clipboard

DNS not working anymore

Open CamJN opened this issue 1 year ago • 14 comments

Description

Trying to pull any new images results in a dns error, despite system dns working fine.

$ docker pull ubuntu:24.04
Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io on 127.0.0.53:53: no such host
$ dscacheutil -q host -a name registry-1.docker.io
name: registry-1.docker.io
ipv6_address: 2600:1f18:2148:bc00:8d61:9b62:40aa:8bb8
ipv6_address: 2600:1f18:2148:bc02:445d:9ace:d20b:c303
ipv6_address: 2600:1f18:2148:bc01:571f:e759:a87a:2961

name: registry-1.docker.io
ip_address: 54.198.86.24
ip_address: 54.236.113.205
ip_address: 54.227.20.253

Version

colima version 0.8.0 git commit: 9c08cff339f087c0600d9d56af7b5fbcfe02e287

runtime: docker arch: x86_64 client: v27.3.1 server: v26.1.1 limactl version 1.0.2

qemu-img: not installed

Operating System

  • [ ] macOS Intel <= 13 (Ventura)
  • [X] macOS Intel >= 14 (Sonoma)
  • [ ] Apple Silicon <= 13 (Ventura)
  • [ ] Apple Silicon >= 14 (Sonoma)
  • [ ] Linux

Output of colima status

$ colima status INFO[0000] colima is running using macOS Virtualization.Framework INFO[0000] arch: x86_64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/camdennarzt/.colima/default/docker.sock

Reproduction Steps

  1. using docker image ls determine an image that you do not have pulled locally
  2. run any docker command which would pull that image eg: docker pull ubuntu:24.04
  3. observe dns being broken

Expected behaviour

DNS should work

Additional context

it seems that there's some issue with lima that might be related: https://github.com/lima-vm/lima/issues/2939

CamJN avatar Dec 04 '24 18:12 CamJN

I have a same issue, today.

jiusi9 avatar Dec 05 '24 09:12 jiusi9

A possible workaround is to assign a reachable address to the virtual machine:

network:
    address: true

jalpedersen avatar Dec 10 '24 11:12 jalpedersen

The same problem is on the ARM64 (M1).

$ colima version
colima version 0.8.0
git commit: 9c08cff339f087c0600d9d56af7b5fbcfe02e287

runtime: docker
arch: aarch64
client: v27.2.1-rd
server: v24.0.9

$ colima status
INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/user/.config/colima/default/docker.sock

Cuchulain avatar Dec 11 '24 15:12 Cuchulain

I ran into that issue as well after updateing to colima 0.8.0 and lima 1.0.2. After some digging we found out that the issue seems to be caused by a DNS proxy as network filter installed by our vpn software. Once that filter is disabled or the vpn client uninstalled DNS started to work again. Everything works if using lima 0.23.2 with colima 0.8.0. The actual change is discussed in the linked issue above. Further experiments imply that the new DNS implementation in lima 1.0.2 via gvisor-tap-vsock which uses Miek Gieben's library https://github.com/miekg/dns is just to fast for that DNS proxy to function. A tiny delay between opening the connection to the DNS server and sending the request makes the filter work. Without the delay the dns request runs in a timeout. I created https://github.com/containers/gvisor-tap-vsock/issues/439 to discuss a change there (knowing that the code there is not the actual problem, rather as a workaround). I would be curious if other people with DNS issues with latest lima also have some network filters running on their macs.

uwej711 avatar Dec 15 '24 14:12 uwej711

Here you can find a simple test to run on your mac: https://github.com/uwej711/test-go-dns

uwej711 avatar Dec 15 '24 14:12 uwej711

And another thing: in my case DNS still works when using TCP instead of UDP, so you can also try to block UDP in ypur virtual machine with iptables:

sudo  iptables -A OUTPUT -p udp -d 192.168.5.2 -j REJECT --reject-with icmp-proto-unreachable

uwej711 avatar Dec 15 '24 14:12 uwej711

I do not use any network filters. The only interesting thing about my dns is that I have dnsmasq installed on my laptop and have macOS use that, rather than the dhcp supplied server.

CamJN avatar Dec 15 '24 18:12 CamJN

Having the same issue. Basically after I get off VPN or on VPN the colima is broken. The only way to solve issue this is to restart it (colima stop && colima start). Of course loosing all running dockers.

colima version
colima version 0.8.1
git commit: 96598cc5b64e5e9e1e64891642b91edc8ac49d16

runtime: docker
arch: aarch64
client: v27.4.1
server: v27.4.1

limactl version 1.0.2

matejsp avatar Dec 21 '24 07:12 matejsp

This is still happening to me, and I do not have a VPN involved at all. Restarting colima does not seem to help, or at least unloading the colima launchd plist and then reloading it doesn't.

CamJN avatar Dec 29 '24 00:12 CamJN

@CamJN do you run Colima with brew services?

abiosoft avatar Jan 01 '25 14:01 abiosoft

Sort of, I use a modified version of the plist from homebrew, but I manage the launchd loading and such myself.

CamJN avatar Jan 01 '25 16:01 CamJN

I run into this problem when tethering to my iPhone on macOS. My work around was to launch Colima as follows: colima start --dns 8.8.8.8 --dns 1.1.1.1.

bradgessler avatar Jan 09 '25 18:01 bradgessler

I run into this problem when tethering to my iPhone on macOS. My work around was to launch Colima as follows: colima start --dns 8.8.8.8 --dns 1.1.1.1.

@bradgessler On a Mac and this worked for me too. Prior to this, a restart of colima didn't resolve it and the only fix was a restart of the Mac. Many thanks!

pricem14pc avatar Feb 14 '25 20:02 pricem14pc

My workaround was to edit /etc/netplan/50-cloud-init.yaml in the lima VM to add an external nameserver, eg.

--- /etc/netplan/50-cloud-init.yaml	2025-04-08 12:08:39.361620679 +1000
+++ /etc/netplan/50-cloud-init.yaml	2025-04-08 10:14:33.399934631 +1000
@@ -6,6 +6,7 @@
         macaddress: "52:55:55:b7:ef:a8"
       nameservers:
         addresses:
+        - 1.1.1.1
         - 192.168.5.2
       dhcp4: true
       dhcp4-overrides:

Then apply that change with netplan apply which allows the new nameserver to take precedence.

asharpe avatar Apr 08 '25 02:04 asharpe