dnsname icon indicating copy to clipboard operation
dnsname copied to clipboard

Published 20 hours ago • verified publisher icon containers

Problem when containers resolve each other by name

Open agdevsu opened this issue 3 years ago • 2 comments

Greetings,

I'm using Podman for containers and the dnsname plugin in order to my Nginx proxy container can resolve the containers name and send requests to the appropriate container. However, sometimes those containers loose their resolution, so the Nginx can't resolve the name and send the request.

Logs generated by Nginx looks like this:

2022/02/16 19:31:42 [error] 8#8: *485534 repuestos_client_1 could not be resolved (2: Server failure), client: 200.7.195.82, server: tems.tde.com.ec, request: "GET /icons/partQuotation.svg HTTP/1.1", host: "tems.tde.com.ec", referrer: "https://tems.tde.com.ec/"
2022/02/16 19:31:43 [error] 8#8: *484842 tracking-angular_tracking_angular_1 could not be resolved (2: Server failure), client: 192.168.11.40, server: tracking.tde.com.ec, request: "GET /sockjs-node/info?t=1645039903403 HTTP/1.1", host: "tracking.tde.com.ec", referrer: "https://tracking.tde.com.ec/"
2022/02/16 19:31:43 [error] 8#8: *485430 repuestos_api_1 could not be resolved (2: Server failure), client: 192.168.8.27, server: tems-api.tde.com.ec, request: "OPTIONS /api/accessories/orders HTTP/1.1", host: "tems-api.tde.com.ec", referrer: "https://tems.tde.com.ec/"
2022/02/16 19:31:43 [error] 8#8: *485040 tracking-angular_tracking_angular_1 could not be resolved (2: Server failure), client: 192.168.6.100, server: tracking.tde.com.ec, request: "GET /sockjs-node/info?t=1645039903892 HTTP/1.1", host: "tracking.tde.com.ec", referrer: "https://tracking.tde.com.ec/"
2022/02/16 19:31:43 [error] 8#8: *485488 repuestos_auth_1 could not be resolved (2: Server failure), client: 200.7.195.82, server: auth.tde.com.ec, request: "GET /auth/realms/Toyota/protocol/openid-connect/logout?redirect_uri=https%3A%2F%2Ftems.tde.com.ec%2F HTTP/1.1", host: "auth.tde.com.ec", referrer: "https://tems.tde.com.ec/"
2022/02/16 19:31:44 [error] 8#8: *485593 repuestos_auth_1 could not be resolved (2: Server failure), client: 10.89.0.1, server: auth.tde.com.ec, request: "POST /auth/realms/Toyota/protocol/openid-connect/token HTTP/1.1", host: "auth.tde.com.ec"
2022/02/16 19:31:44 [error] 8#8: *482445 tracking-angular_tracking_angular_1 could not be resolved (2: Server failure), client: 192.168.8.27, server: tracking.tde.com.ec, request: "GET /sockjs-node/info?t=1645039904665 HTTP/1.1", host: "tracking.tde.com.ec", referrer: "https://tracking.tde.com.ec/"
2022/02/16 19:31:45 [error] 8#8: *477697 tracking-angular_tracking_angular_1 could not be resolved (2: Server failure), client: 192.168.3.3, server: tracking.tde.com.ec, request: "GET /sockjs-node/info?t=1645039446665 HTTP/1.1", host: "tracking.tde.com.ec", referrer: "https://tracking.tde.com.ec/?nuevaUbicacion=H-9&observacion="
2022/02/16 19:31:45 [error] 8#8: *485594 repuestos_auth_1 could not be resolved (2: Server failure), client: 157.100.97.2, server: auth.tde.com.ec, request: "POST /auth/realms/Toyota/protocol/openid-connect/token HTTP/1.1", host: "auth.tde.com.ec", referrer: "https://tems.tde.com.ec/"
2022/02/16 19:31:45 [error] 8#8: *485594 repuestos_auth_1 could not be resolved (2: Server failure), client: 157.100.97.2, server: auth.tde.com.ec, request: "GET /auth/realms/Toyota/protocol/openid-connect/logout?redirect_uri=https%3A%2F%2Ftems.tde.com.ec%2Fconcesionarios%2Fpedidos-emergentes HTTP/1.1", host: "auth.tde.com.ec", referrer: "https://tems.tde.com.ec/"
2022/02/16 19:31:45 [error] 8#8: *475349 tracking-angular_tracking_angular_1 could not be resolved (2: Server failure), client: 10.10.10.13, server: tracking.tde.com.ec, request: "GET /sockjs-node/info?t=1645040053465 HTTP/1.1", host: "tracking.tde.com.ec", referrer: "https://tracking.tde.com.ec/"

Now, /run/containers/cni/dnsname/repuestos_default/addhosts file looks like this: Screenshot from 2022-02-17 10-29-11

I don't know why this happens. If I reboot the server, the resolution works again.

Output from podman info:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-1.module+el8.5.0+12582+56d94c81.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: 0f5bee61b18d4581668e5bf18b910cda3cff5081'
  cpus: 4
  distribution:
    distribution: '"rhel"'
    version: "8.5"
  eventLogger: file
  hostname: tdepodmanpd
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-348.2.1.el8_5.x86_64
  linkmode: dynamic
  memFree: 164163584
  memTotal: 12372725760
  ociRuntime:
    name: runc
    package: runc-1.0.2-1.module+el8.5.0+12582+56d94c81.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2
      spec: 1.0.2-dev
      go: go1.16.7
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.5.0+12582+56d94c81.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 10166218752
  swapTotal: 12884897792
  uptime: 20h 17m 32.36s (Approximately 0.83 days)
registries:
  192.168.3.236:5000:
    Blocked: false
    Insecure: true
    Location: 192.168.3.236:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: 192.168.3.236:5000
  192.168.3.238:5000:
    Blocked: false
    Insecure: true
    Location: 192.168.3.238:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: 192.168.3.238:5000
  search:
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 15
    paused: 0
    running: 15
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 26
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1632213702
  BuiltTime: Tue Sep 21 03:41:42 2021
  GitCommit: ""
  GoVersion: go1.16.7
  OsArch: linux/amd64
  Version: 3.3.1

Thanks for help!

agdevsu avatar Feb 17 '22 15:02 agdevsu

i notice that your distro is RHEL. Can you please take this to bugzilla?

baude avatar Feb 17 '22 16:02 baude

Based on the description of @agdevsu I seem to have a similar problem on openSuse MicroOS. I have two pods and the resolution works sometimes on my website and sometimes on my ticketing system, never on both - sometimes osticket.dns.osticket works and sometimes joomla.dns.podman works (logs from nginx podman):

2023/01/23 17:57:59 [emerg] 1#1: host not found in upstream "joomla.dns.podman" in /etc/nginx/conf.d/example.com.conf:27
nginx: [emerg] host not found in upstream "joomla.dns.podman" in /etc/nginx/conf.d/example.com.conf:27
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/01/23 17:58:01 [emerg] 1#1: host not found in upstream "osticket.dns.osticket" in /etc/nginx/conf.d/support.example.com.conf:22
nginx: [emerg] host not found in upstream "osticket.dns.osticket" in /etc/nginx/conf.d/support.example.com.conf:22
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/01/23 17:58:04 [emerg] 1#1: host not found in upstream "joomla.dns.podman" in /etc/nginx/conf.d/example.com.conf:27
nginx: [emerg] host not found in upstream "joomla.dns.podman" in /etc/nginx/conf.d/example.com.conf:27
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/01/23 17:58:07 [emerg] 1#1: host not found in upstream "osticket.dns.osticket" in /etc/nginx/conf.d/support.example.com.conf:22
nginx: [emerg] host not found in upstream "osticket.dns.osticket" in /etc/nginx/conf.d/support.example.com.conf:22

I'd be really glad if someone could nudge me in the right direction, because I've got no clue where the issue could lie.

podman info:

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.5-2.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: unknown'
  cpuUtilization:
    idlePercent: 95.71
    systemPercent: 2.42
    userPercent: 1.87
  cpus: 2
  distribution:
    distribution: '"opensuse-microos"'
    version: "20230119"
  eventLogger: journald
  hostname: pvsr-laweb08
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.7-1-default
  linkmode: dynamic
  logDriver: journald
  memFree: 2121355264
  memTotal: 4113506304
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.1.4-2.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.4
      commit: v1.1.4-0-ga916309fff0f
      spec: 1.0.2-dev
      go: go1.18.6
      libseccomp: 2.5.4
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-1.1.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 0h 17m 32.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - docker.io
store:
  configFile: /home/admin/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 6
    stopped: 1
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /home/admin/.local/share/containers/storage
  graphRootAllocated: 21450698752
  graphRootUsed: 12955815936
  graphStatus:
    Build Version: Btrfs v6.1.2
    Library Version: "102"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 13
  runRoot: /run/user/1000/containers
  volumePath: /home/admin/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1673913600
  BuiltTime: Tue Jan 17 01:00:00 2023
  GitCommit: ""
  GoVersion: go1.17.13
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

fipoac avatar Jan 23 '23 17:01 fipoac