Cannot resolve container hostnames in internal network

[jorti]

Issue Description

The DNS resolution in an internal network doesn't work. The DNS server is unreachable.

Steps to reproduce the issue

Steps to reproduce the issue. Run all commands as root:

1. Create internal network:

# podman network create internal-network --internal

2. Create first container:

# podman run -d --rm --name container1 --network internal-network fedora sleep infinity

3. In a second container, try to resolve the first container hostname:

# podman run -ti --rm --network internal-network quay.io/jorti/debug
[root@5823c479811c /]# cat /etc/resolv.conf 
search dns.podman
nameserver 10.89.2.1
[root@5823c479811c /]# dig container1
;; communications error to 10.89.2.1#53: host unreachable
;; communications error to 10.89.2.1#53: host unreachable
;; communications error to 10.89.2.1#53: host unreachable

; <<>> DiG 9.18.26 <<>> container1
;; global options: +cmd
;; no servers could be reached

Describe the results you received

Internal DNS server is unreachable. I cannot resolve other container hostnames in an internal network.

Describe the results you expected

This network uses the netavark backend and the bridge driver, so I expect a working DNS functionality.

podman info output

host:
  arch: amd64
  buildahVersion: 1.35.4
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 86.94
    systemPercent: 2.59
    userPercent: 10.47
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: workstation
    version: "40"
  eventLogger: journald
  freeLocks: 2045
  hostname: _REDACTED_
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.8.10-300.fc40.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 6752358400
  memTotal: 33483927552
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240510.g7288448-1.fc40.x86_64
    version: |
      pasta 0^20240510.g7288448-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 1h 39m 8.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 1
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 474386456576
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 9
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.0.3
  Built: 1715299200
  BuiltTime: Fri May 10 02:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.3

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

[Luap99]

Do you have firewalld running, or other firewall rules configured?

The thing for internal networks is that we do not configure any firewall rules at all as they are internal. However it seems (at least) firewalld is blocking the dns quires by default so it cannot connect to the aardvark-dns instance.

For firewalld we need to set it trusted like we do for normal non internal networks firewall-cmd --zone=trusted --add-source <subnet> or add a specific rule to only allow dns from the container.

cc @mheon

[jorti]

Do you have firewalld running, or other firewall rules configured?

Yes, I have firewalld using the default settings from Fedora 40 Server.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[flouthoc]

@Luap99 Is this still an issue ? I have firewalld enabled and for me this works on --internal network.

[flouthoc]

@Luap99 Is this still an issue ? I have firewalld enabled and for me this works on --internal network.

Nvm i think my driver is being selected as iptables even though firewalld is running :, well i can confirm this works with iptables

[Luap99]

@Luap99 Is this still an issue ? I have firewalld enabled and for me this works on --internal network.

Nvm i think my driver is being selected as iptables even though firewalld is running :, well i can confirm this works with iptables

No it doesn't, in general an internal network doesn't create any firewall rule whatsoever currently as pointed out above as such the driver doesn't matter. The issue is that if your host has a rule to drop all traffic by default then this will break our dns queries (that may be set by firewalld or any other user). We explicitly add an accept rule for out dns traffic for normal networks but as internal ones do not use the firewall driver this is not added there.

We certainly must fix that to always add such rule if dns is enabled. This is also the reason for https://github.com/containers/netavark/issues/1051.

I will transfer the issue to netavark as it must be fixed there.

[flouthoc]

@Luap99 I tried this on my machine , rootless

podman network create test-internal --internal

and

podman run -it --rm --network=test-internal --name ctr1 nicolaka/netshoot bash
81fc4548a449:~# dig ctr1

; <<>> DiG 9.18.25 <<>> ctr1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52951
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: c6de17eea74b9b14 (echoed)
;; QUESTION SECTION:
;ctr1.				IN	A

;; ANSWER SECTION:
ctr1.			60	IN	A	10.89.1.4

;; Query time: 0 msec
;; SERVER: 10.89.1.1#53(10.89.1.1) (UDP)
;; WHEN: Mon Aug 12 13:46:42 UTC 2024
;; MSG SIZE  rcvd: 61

also tried resolution between two containers on internal network and it worked just fine, i am not using firewalld.

[Luap99]

yes because you have a default accept rule, change your firewall to default drop and it will no longer work. https://github.com/containers/netavark/issues/742

[flouthoc]

yes because you have a default accept rule, change your firewall to default drop and it will no longer work. #742

That must be it, i have a default accept rule.