resetting user-mode-networking cause dns issues

[jeffmaury]

Issue Description

I created a podman image, set user-mode-networking and started it. Pulling an image is ok. Then stop it; reset the user-mode-networking, restart it: pull images is rejected

Steps to reproduce the issue

Steps to reproduce the issue

1. podman machine init
2. podman machine set user-mode-networking
3. podman machine start
4. podman pull httpd
5. podman machine stop
6. podman machine set --user-mode-networking=false
7. podman machine start
8. podman pull alpine

Describe the results you received

$ podman pull alpine Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) Trying to pull docker.io/library/alpine:latest... Error: initializing source docker://alpine:latest: pinging container registry registry-1.docker.io: Get "https://registry-1.docker.io/v2/": dial tcp: lookup registry-1.docker.io: Temporary failure in name resolution

Describe the results you expected

alpine image should be pulled

podman info output

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 99.73
    systemPercent: 0.13
    userPercent: 0.14
  cpus: 12
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: container
    version: "38"
  eventLogger: journald
  freeLocks: 2048
  hostname: DESKTOP-JEFF
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.133.1-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 16073285632
  memTotal: 16646295552
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11.2-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.11.2
      commit: ab0edeef1c331840b025e8f1d38090cfb8a0509d
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231107.g56d9f6d-1.fc38.x86_64
    version: |
      pasta 0^20231107.g56d9f6d-1.fc38.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc38.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 4294967296
  swapTotal: 4294967296
  uptime: 0h 7m 29.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 807157760
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 1698762611
  BuiltTime: Tue Oct 31 15:30:11 2023
  GitCommit: ""
  GoVersion: go1.20.10
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[jsearby]

Hello

I had similar issue (using podman on windows) just after doing

podman machine stop
podman machine set --user-mode-networking=false
podman machine start

For my case , root cause was "an invalid config" pushed into /etc/resolv.conf (inside the podman WSL distribution). It was displaying a single line (nameserver x.x.x.x) pointing to the WSL interface IP (which is wrong)

I'm not sure what from above steps causes a wrong edition of /etc/resolv.conf .. but for sure it breaks all the DNS resolution, making network no more usable.

Note1: Just re-editing the /etc/resolv.conf worked for me

#From windows host
wsl --list --verbose
wsl --distribution <<The podman machine you are using>>
#From the WSL linux (where we just jumped into)
sudo vi etc/resolv.conf
#Then add a valid nameserver IP (get it from your windows one)

Note2: Normally the /etc/wsl.conf can be edited to explicit "generateResolvConf = false" Such should avoid re-configuration of the /etc/resolv.conf. When missing, it could worth that podman ensure the nameserver written in /etc/resolv.conf is at least a valid one (in our case "main computer default DNS server" would be a far better choice than the WSL interface IP)

If, like me, you run podman from windows , the actual default DNS server can be found using following powershell comand Get-DnsClientServerAddress -AddressFamily IPv4 | Select-Object -ExpandProperty ServerAddresses

[NicolaiLolansen]

@jsearby Thank you so much for this answer! I had to reinstall Podman into v4.9.0 and I had the exact same experience. I'm behind a corporate network, so it's important the DNS stays the same. Not sure what changed since last, but I'm quite sure this will break for many other users as well.

Your fix worked for me, little cumbersome you have to edit it through vi but it is what it is.

[Luap99]

Can you retest with podman 5.0?

[jeffmaury]

See #24061 for a reproducer on Podman 5.2.2

[NicolaiLolansen]

It's still an issue for me as well, and it breaks every time I have to re-create the VM