docker-alpine coredns 1.8.3 not able to resolve records which has CNAME/ALIAS on alpine 3.9.6, 3.12, 3.16.1

coredns 1.8.3 not able to resolve records which has CNAME/ALIAS on alpine 3.9.6, 3.12, 3.16.1 - curl/wget

Open kranthirachakonda opened this issue 1 year ago • 7 comments

I m having a kind of weird issue with resolution with curl/wget of an endpoint on our pods/rancher k8s-cluster (1.20), which is using coredns-1.8.0 (& 1.8.3) and alpine 3.9.6, 3.12, 3.16.1 (i tried all versions of 3.9+). As most of the other dns issues filed earlier, same resolution works on local mac and VM/Nodes using same NS servers. Other public records like github.com, api.twilio.com they resolve correctly.

I tried below and also read about known issues with alpine dns resolution issues, but not sure if I am running into a known issue or something wrong with my setup. Need help resolving it.

Record which is having issue (changed hostname, its not correct record):

dev.apim.sample-dns.com is an alias for sample-dns-apim-dev.azure-api.net. sample-dns-apim-dev.azure-api.net is an alias for abc123459cgudpyyjn4c34vuw.trafficmanager.net. abc123459cgudpyyjn4c34vuw.trafficmanager.net is an alias for sample-dns-apim-dev-eastus-01.regional.azure-api.net. sample-dns-apim-dev-eastus-01.regional.azure-api.net is an alias for abcdns1234564ef174badlkbkiw0s6xvttwcjjn90n.eastus.cloudapp.azure.com. abcdns1234564ef174badlkbkiw0s6xvttwcjjn90n.eastus.cloudapp.azure.com has address 1xx.2x.3x.4

On the pod:

/ # curl -vk dev.apim.sample-dns.com
* Could not resolve host: dev.apim.sample-dns.com
* Closing connection 0
curl: (6) Could not resolve host: dev.apim.sample-dns.com
/ # nslookup dev.apim.sample-dns.com
Server:         10.3.0.10
Address:        10.3.0.10#53


dev.apim.sample-dns.com is an alias for sample-dns-apim-dev.azure-api.net.
sample-dns-apim-dev.azure-api.net is an alias for abc123459cgudpyyjn4c34vuw.trafficmanager.net.
abc123459cgudpyyjn4c34vuw.trafficmanager.net is an alias for sample-dns-apim-dev-eastus-01.regional.azure-api.net.
sample-dns-apim-dev-eastus-01.regional.azure-api.net is an alias for abcdns1234564ef174badlkbkiw0s6xvttwcjjn90n.eastus.cloudapp.azure.com.
Name:   abcdns1234564ef174badlkbkiw0s6xvttwcjjn90n.eastus.cloudapp.azure.com
Address: 1xx.2x.3x.4

/ # cat /etc/resolv.conf 
nameserver 10.3.0.10
search default.svc.cluster.local svc.cluster.local cluster.local sample-dns.com 123.sampledns.com
options ndots:5
/ # 


Tried these which didnt help, i.e. curl/wget doesnt resolve above record:

Reduced ndots to 3
Initially nslookup was failing to resolve and giving some random public ips. Read somewhere installing bind-tools helps.

/ # nslookup dev.apim.sample-dns.com
nslookup: can't resolve '(null)': Name does not resolve


Name:      dev.apim.sample-dns.com
Address 1: 93.239.133.128 p5def8580.dip0.t-ipconnect.de

Installed bind-tools as apk add bind-tools for test. And that helped nslookup to resolve with correct/expected ip-address and dns name

Took a tcpdump and see that traffic from pod to coredns/kube-dns.kube-system.svc.cluster.local and there to our infoblox server. In response, I do see correct aliases, cname, A record in tcpdump. However response on my resolution shows random public ips.
curl and wget resolve to different ips. Not sure why
nslookup & dig resolve to correct/expected ip.

corefile:
.:53 {
    log
    health {
      lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
      pods insecure
      fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . "/etc/resolv.conf"
    bufsize 1232 << added this for other records which were >512B
    cache 30
    loop
    reload
    loadbalance
} # STUBDOMAINS - Rancher specific change

Please let me know if there are any questions or need additional info.

Oct 04 '22 15:10 kranthirachakonda

coredns logs

[INFO] 10.2.9.142:55395 - 33583 "AAAA IN dev.apim.sample-dns.com.lowes.com. udp 53 false 1232" NXDOMAIN qr,aa,rd,ra 127 0.000153288s [INFO] 10.2.9.142:55395 - 33258 "A IN dev.apim.sample-dns.com.lowes.com. udp 53 false 1232" NXDOMAIN qr,aa,rd,ra 127 0.000126996s [INFO] 10.2.9.142:39655 - 51233 "AAAA IN dev.apim.sample-dns.com. udp 43 false 1232" NOERROR qr,aa,rd,ra 526 0.000095743s [INFO] 10.2.9.142:39655 - 51038 "A IN dev.apim.sample-dns.com. udp 43 false 1232" NOERROR qr,aa,rd,ra 854 0.00173528s

Oct 04 '22 16:10 kranthirachakonda

Hi there.

Still trying to get all the details around a similar issue.

We are using Kubernetes clusters on Azure. (coredns 1.8.7) And our base images are Alpine images. We tested various 3.15 and 3.16 images.

DNS resolution often fails on alpine based pods. We tried debian and ubuntu based images within the same clusters (and same coredns). and name resolution is fine.

Some posts suggests this is due to the lack of TCP support in the MUSL library (https://gitlab.alpinelinux.org/alpine/tsc/-/issues/43)

Behaviour :

/ # cat /etc/resolv.conf
search <any-namespace>.svc.cluster.local svc.cluster.local cluster.local 
nameserver 10.132.0.10   # kube-dns svc ip
options ndots:5
/ # curl https://gitlab.com
curl: (6) Could not resolve host: gitlab.com
/ # curl https://gitlab.com.
<html><body>You are being <a href="https://about.gitlab.com/">redirected</a>.</body></html>/ #

Oct 07 '22 13:10 keisari-ch

Posting a strace if that could be helpful for issue.

bash-5.1# strace nslookup gitlab.com
execve("/usr/bin/nslookup", ["nslookup", "gitlab.com"], 0x7ffebb9b8238 /* 54 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x7f7e9c865b48) = 0
set_tid_address(0x7f7e9c865fb0)         = 607
brk(NULL)                               = 0x558828d99000
brk(0x558828d9b000)                     = 0x558828d9b000
mmap(0x558828d99000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x558828d99000
open("/etc/ld-musl-x86_64.path", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libdns-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libdns-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libdns-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=1886952, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 1892352, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c601000
mmap(0x7f7e9c62e000, 1241088, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x2d000) = 0x7f7e9c62e000
mmap(0x7f7e9c75d000, 421888, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x15c000) = 0x7f7e9c75d000
mmap(0x7f7e9c7c4000, 45056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1c2000) = 0x7f7e9c7c4000
close(3)                                = 0
open("/lib/libirs-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libirs-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libirs-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=42832, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 49152, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c5f5000
mmap(0x7f7e9c5f8000, 20480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x3000) = 0x7f7e9c5f8000
mmap(0x7f7e9c5fd000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x8000) = 0x7f7e9c5fd000
mmap(0x7f7e9c5ff000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x9000) = 0x7f7e9c5ff000
close(3)                                = 0
open("/lib/libbind9-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libbind9-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libbind9-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=79608, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 86016, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c5e0000
mmap(0x7f7e9c5e3000, 45056, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x3000) = 0x7f7e9c5e3000
mmap(0x7f7e9c5ee000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0xe000) = 0x7f7e9c5ee000
mmap(0x7f7e9c5f3000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x12000) = 0x7f7e9c5f3000
close(3)                                = 0
open("/lib/libisccfg-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libisccfg-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libisccfg-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=198008, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 208896, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c5ad000
mmap(0x7f7e9c5c1000, 49152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x14000) = 0x7f7e9c5c1000
mmap(0x7f7e9c5cd000, 28672, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x20000) = 0x7f7e9c5cd000
mmap(0x7f7e9c5d5000, 45056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x27000) = 0x7f7e9c5d5000
mmap(0x7f7e9c5de000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7e9c5de000
close(3)                                = 0
open("/lib/libisc-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libisc-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libisc-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=462824, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 471040, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c53a000
mmap(0x7f7e9c54f000, 241664, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x15000) = 0x7f7e9c54f000
mmap(0x7f7e9c58a000, 122880, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x50000) = 0x7f7e9c58a000
mmap(0x7f7e9c5a9000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x6e000) = 0x7f7e9c5a9000
mmap(0x7f7e9c5ac000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7e9c5ac000
close(3)                                = 0
open("/lib/libgssapi_krb5.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libgssapi_krb5.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libgssapi_krb5.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=291144, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 294912, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c4f2000
mmap(0x7f7e9c4fe000, 180224, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0xc000) = 0x7f7e9c4fe000
mmap(0x7f7e9c52a000, 53248, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x38000) = 0x7f7e9c52a000
mmap(0x7f7e9c537000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x44000) = 0x7f7e9c537000
close(3)                                = 0
open("/lib/libkrb5.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libkrb5.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libkrb5.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=805488, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 811008, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c42c000
mmap(0x7f7e9c44f000, 307200, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x23000) = 0x7f7e9c44f000
mmap(0x7f7e9c49a000, 290816, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x6e000) = 0x7f7e9c49a000
mmap(0x7f7e9c4e1000, 69632, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xb4000) = 0x7f7e9c4e1000
close(3)                                = 0
open("/lib/libfstrm.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libfstrm.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libfstrm.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=42904, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 49152, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c420000
mmap(0x7f7e9c423000, 16384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x3000) = 0x7f7e9c423000
mmap(0x7f7e9c427000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x7000) = 0x7f7e9c427000
mmap(0x7f7e9c42a000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x9000) = 0x7f7e9c42a000
close(3)                                = 0
open("/lib/libprotobuf-c.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libprotobuf-c.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libprotobuf-c.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=30568, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 36864, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c417000
mmap(0x7f7e9c418000, 16384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x1000) = 0x7f7e9c418000
mmap(0x7f7e9c41c000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x5000) = 0x7f7e9c41c000
mmap(0x7f7e9c41e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x6000) = 0x7f7e9c41e000
close(3)                                = 0
open("/lib/libns-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libns-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libns-9.16.33.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=254288, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 258048, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c3d8000
mmap(0x7f7e9c3e2000, 155648, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0xa000) = 0x7f7e9c3e2000
mmap(0x7f7e9c408000, 45056, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x30000) = 0x7f7e9c408000
mmap(0x7f7e9c414000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x3b000) = 0x7f7e9c414000
close(3)                                = 0
open("/lib/libssl.so.1.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=523704, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 528384, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c357000
mmap(0x7f7e9c373000, 262144, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x1c000) = 0x7f7e9c373000
mmap(0x7f7e9c3b3000, 98304, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x5c000) = 0x7f7e9c3b3000
mmap(0x7f7e9c3cb000, 53248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x73000) = 0x7f7e9c3cb000
close(3)                                = 0
open("/lib/libcrypto.so.1.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=2609840, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 2629632, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c0d5000
mmap(0x7f7e9c14a000, 1404928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x75000) = 0x7f7e9c14a000
mmap(0x7f7e9c2a1000, 544768, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x1cc000) = 0x7f7e9c2a1000
mmap(0x7f7e9c326000, 200704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x250000) = 0x7f7e9c326000
mmap(0x7f7e9c353000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7e9c353000
close(3)                                = 0
open("/lib/libjson-c.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libjson-c.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libjson-c.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=59512, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 65536, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c0c5000
mmap(0x7f7e9c0c9000, 28672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x4000) = 0x7f7e9c0c9000
mmap(0x7f7e9c0d0000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0xb000) = 0x7f7e9c0d0000
mmap(0x7f7e9c0d3000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xd000) = 0x7f7e9c0d3000
close(3)                                = 0
open("/lib/libuv.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libuv.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libuv.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=173896, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 180224, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9c099000
mmap(0x7f7e9c0a3000, 90112, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0xa000) = 0x7f7e9c0a3000
mmap(0x7f7e9c0b9000, 36864, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x20000) = 0x7f7e9c0b9000
mmap(0x7f7e9c0c3000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x29000) = 0x7f7e9c0c3000
close(3)                                = 0
open("/lib/libxml2.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libxml2.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxml2.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=1213568, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 1220608, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9bf6f000
mmap(0x7f7e9bf9e000, 708608, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x2f000) = 0x7f7e9bf9e000
mmap(0x7f7e9c04b000, 274432, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0xdc000) = 0x7f7e9c04b000
mmap(0x7f7e9c08e000, 45056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x11e000) = 0x7f7e9c08e000
mmap(0x7f7e9c098000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7e9c098000
close(3)                                = 0
open("/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=100248, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 106496, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9bf55000
mmap(0x7f7e9bf58000, 57344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x3000) = 0x7f7e9bf58000
mmap(0x7f7e9bf66000, 28672, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x11000) = 0x7f7e9bf66000
mmap(0x7f7e9bf6d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x17000) = 0x7f7e9bf6d000
close(3)                                = 0
open("/lib/libk5crypto.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libk5crypto.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libk5crypto.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=178376, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 184320, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9bf28000
mmap(0x7f7e9bf2c000, 102400, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x4000) = 0x7f7e9bf2c000
mmap(0x7f7e9bf45000, 53248, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x1d000) = 0x7f7e9bf45000
mmap(0x7f7e9bf52000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x29000) = 0x7f7e9bf52000
close(3)                                = 0
open("/lib/libcom_err.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=14080, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 20480, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9bf23000
mmap(0x7f7e9bf24000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x1000) = 0x7f7e9bf24000
mmap(0x7f7e9bf25000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x7f7e9bf25000
mmap(0x7f7e9bf26000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x7f7e9bf26000
close(3)                                = 0
open("/lib/libkrb5support.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libkrb5support.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libkrb5support.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=47296, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 53248, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9bf16000
mmap(0x7f7e9bf19000, 20480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x3000) = 0x7f7e9bf19000
mmap(0x7f7e9bf1e000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x8000) = 0x7f7e9bf1e000
mmap(0x7f7e9bf21000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xa000) = 0x7f7e9bf21000
close(3)                                = 0
open("/lib/libkeyutils.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libkeyutils.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libkeyutils.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=22344, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 28672, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9bf0f000
mmap(0x7f7e9bf11000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x7f7e9bf11000
mmap(0x7f7e9bf13000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x4000) = 0x7f7e9bf13000
mmap(0x7f7e9bf14000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x4000) = 0x7f7e9bf14000
close(3)                                = 0
open("/lib/liblzma.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/liblzma.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/liblzma.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=136952, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 143360, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e9beec000
mmap(0x7f7e9beef000, 77824, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x3000) = 0x7f7e9beef000
mmap(0x7f7e9bf02000, 45056, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x16000) = 0x7f7e9bf02000
mmap(0x7f7e9bf0d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x20000) = 0x7f7e9bf0d000
close(3)                                = 0
mprotect(0x7f7e9c7c4000, 24576, PROT_READ) = 0
mprotect(0x7f7e9c5ff000, 4096, PROT_READ) = 0
mprotect(0x7f7e9c5f3000, 4096, PROT_READ) = 0
mprotect(0x7f7e9c5d5000, 4096, PROT_READ) = 0
mprotect(0x7f7e9c5a9000, 8192, PROT_READ) = 0
mprotect(0x7f7e9c862000, 4096, PROT_READ) = 0
mprotect(0x7f7e9c537000, 8192, PROT_READ) = 0
mprotect(0x7f7e9c4e1000, 61440, PROT_READ) = 0
mprotect(0x7f7e9c42a000, 4096, PROT_READ) = 0
mprotect(0x7f7e9c41e000, 4096, PROT_READ) = 0
mprotect(0x7f7e9c414000, 8192, PROT_READ) = 0
mprotect(0x7f7e9c3cb000, 36864, PROT_READ) = 0
mprotect(0x7f7e9c326000, 176128, PROT_READ) = 0
mprotect(0x7f7e9c0d3000, 4096, PROT_READ) = 0
mprotect(0x7f7e9c0c3000, 4096, PROT_READ) = 0
mprotect(0x7f7e9c08e000, 36864, PROT_READ) = 0
mprotect(0x7f7e9bf6d000, 4096, PROT_READ) = 0
mprotect(0x7f7e9bf52000, 8192, PROT_READ) = 0
mprotect(0x7f7e9bf26000, 4096, PROT_READ) = 0
mprotect(0x7f7e9bf21000, 4096, PROT_READ) = 0
mprotect(0x7f7e9bf14000, 4096, PROT_READ) = 0
mprotect(0x7f7e9bf0d000, 4096, PROT_READ) = 0
mprotect(0x55882898d000, 4096, PROT_READ) = 0
arch_prctl(ARCH_SET_FS, 0x7f7e9c866490) = 0
set_tid_address(0x7f7e9c865fb0)         = 607
brk(0x558828d9c000)                     = 0x558828d9c000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9beeb000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bee9000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bee8000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bee7000
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bee3000
munmap(0x7f7e9bee9000, 8192)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9beea000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bee9000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bee1000
mmap(NULL, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9beda000
munmap(0x7f7e9bee3000, 16384)           = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bee5000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bee3000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bed8000
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bed4000
mmap(NULL, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9becf000
madvise(0x7f7e9bedb000, 4096, MADV_FREE) = 0
munmap(0x7f7e9beda000, 28672)           = 0
brk(0x558828d9d000)                     = 0x558828d9d000
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bedd000
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
open("/etc/ssl/openssl.cnf", O_RDONLY|O_LARGEFILE) = 3
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bedc000
read(3, "#\n# OpenSSL example configuratio"..., 1024) = 1024
read(3, "################################"..., 1024) = 1024
read(3, "\t= ca_default\t\t# Subject Name op"..., 1024) = 1024
read(3, "me, you must list all acceptable"..., 1024) = 1024
read(3, "# WARNING: ancient versions of N"..., 1024) = 1024
read(3, "butes ]\nchallengePassword\t\t= A c"..., 1024) = 1024
read(3, "L Generated Certificate\"\n\n# PKIX"..., 1024) = 1024
read(3, "ver since it will\n# prevent it b"..., 1024) = 1024
read(3, "es of the usage of nsCertType. I"..., 1024) = 1024
read(3, "issuer:copy\n\n#nsCaRevocationUrl\t"..., 1024) = 1024
read(3, "onal)\nother_policies\t= tsa_polic"..., 1024) = 669
read(3, "", 1024)                       = 0
munmap(0x7f7e9bedc000, 4096)            = 0
close(3)                                = 0
getpid()                                = 607
getrandom("\x3c\xfd\xf5\x6b\x4b\xe0\x33\xf1\x64\x38\xa2\x69\x77\xc1\x2a\x0d\x88\xf3\x9d\xc1\xe6\x39\xfc\x23\x69\x58\x2a\x71\x7f\x43\x68\x6e", 32, 0) = 32
getpid()                                = 607
ioctl(0, TIOCGWINSZ, {ws_row=61, ws_col=269, ws_xpixel=0, ws_ypixel=0}) = 0
rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER, sa_restorer=0x7f7e9c815c9f}, NULL, 8) = 0
rt_sigaction(SIGHUP, {sa_handler=SIG_DFL, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER, sa_restorer=0x7f7e9c815c9f}, NULL, 8) = 0
rt_sigaction(SIGTERM, {sa_handler=SIG_DFL, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER, sa_restorer=0x7f7e9c815c9f}, NULL, 8) = 0
rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=~[RTMIN RT_1 RT_2], sa_flags=SA_RESTORER, sa_restorer=0x7f7e9c815c9f}, NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [HUP INT TERM], NULL, 8) = 0
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
close(3)                                = 0
socket(AF_INET6, SOCK_STREAM, IPPROTO_IP) = 3
getsockname(3, {sa_family=AF_INET6, sin6_port=htons(0), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "::", &sin6_addr), sin6_scope_id=0}, [28]) = 0
close(3)                                = 0
socket(AF_UNIX, SOCK_STREAM, 0)         = 3
close(3)                                = 0
mmap(NULL, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bec6000
mmap(NULL, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bebf000
clock_getres(CLOCK_MONOTONIC_COARSE, {tv_sec=0, tv_nsec=4000000}) = 0
epoll_create1(EPOLL_CLOEXEC)            = 3
pipe2([4, 5], O_CLOEXEC)                = 0
write(5, "*", 1)                        = 1
pipe2([6, 7], O_NONBLOCK|O_CLOEXEC)     = 0
eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK)   = 8
mmap(NULL, 1310721, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bd7e000
mmap(NULL, 69632, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bd6d000
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
membarrier(MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED, 0) = 0
mmap(NULL, 1060864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bc6a000
mprotect(0x7f7e9bc6c000, 1052672, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT TERM], 8) = 0
clone(child_stack=0x7f7e9bd6ca58, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tid=[608], tls=0x7f7e9bd6cb38, child_tidptr=0x7f7e9c865fb0) = 608
rt_sigprocmask(SIG_SETMASK, [HUP INT TERM], NULL, 8) = 0
open("/proc/self/task/608/comm", O_WRONLY|O_LARGEFILE|O_CLOEXEC) = 9
fcntl(9, F_SETFD, FD_CLOEXEC)           = 0
write(9, "isc-net-0000", 12)            = 12
close(9)                                = 0
mmap(NULL, 1060864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bb67000
mprotect(0x7f7e9bb69000, 1052672, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT TERM], 8) = 0
clone(child_stack=0x7f7e9bc69a58, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tid=[609], tls=0x7f7e9bc69b38, child_tidptr=0x7f7e9c865fb0) = 609
rt_sigprocmask(SIG_SETMASK, [HUP INT TERM], NULL, 8) = 0
open("/proc/self/task/609/comm", O_WRONLY|O_LARGEFILE|O_CLOEXEC) = 9
fcntl(9, F_SETFD, FD_CLOEXEC)           = 0
write(9, "isc-timer", 9)                = 9
close(9)                                = 0
mmap(NULL, 168021, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bb3d000
mmap(NULL, 86016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bb28000
mmap(NULL, 45056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bb1d000
pipe([9, 10])                           = 0
fcntl(9, F_GETFL)                       = 0 (flags O_RDONLY)
fcntl(9, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 0
mmap(NULL, 86016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bb08000
mmap(NULL, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bb01000
epoll_create1(0)                        = 11
epoll_ctl(11, EPOLL_CTL_ADD, 9, {events=EPOLLIN, data={u32=9, u64=9}}) = 0
mmap(NULL, 1060864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9b9fe000
mprotect(0x7f7e9ba00000, 1052672, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT TERM], 8) = 0
clone(child_stack=0x7f7e9bb00a58, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tid=[610], tls=0x7f7e9bb00b38, child_tidptr=0x7f7e9c865fb0) = 610
rt_sigprocmask(SIG_SETMASK, [HUP INT TERM], NULL, 8) = 0
open("/proc/self/task/610/comm", O_WRONLY|O_LARGEFILE|O_CLOEXEC) = 12
fcntl(12, F_SETFD, FD_CLOEXEC)          = 0
write(12, "isc-socket-0", 12)           = 12
close(12)                               = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bedc000
munmap(0x7f7e9bedc000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bedc000
munmap(0x7f7e9bedc000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bedc000
munmap(0x7f7e9bedc000, 4096)            = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bedc000
munmap(0x7f7e9bedc000, 4096)            = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9bedb000
open("/etc/resolv.conf", O_RDONLY|O_LARGEFILE) = 12
read(12, "search kube-system.svc.cluster.l"..., 1024) = 129
read(12, "", 1024)                      = 0
close(12)                               = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9beda000
getpid()                                = 607
getpid()                                = 607
getpid()                                = 607
getpid()                                = 607
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e9b9fa000
getpid()                                = 607
getpid()                                = 607
write(8, "\1\0\0\0\0\0\0\0", 8)         = 8
rt_sigtimedwait([HUP INT TERM], Server:         10.132.0.10
Address:        10.132.0.10#53

*** Can't find gitlab.com.reddog.microsoft.com: No answer
{si_signo=SIGTERM, si_code=SI_USER, si_pid=607, si_uid=0}, NULL, 8) = 15 (SIGTERM)
writev(1, [{iov_base="", iov_len=0}, {iov_base="\n", iov_len=1}], 2
) = 1
write(8, "\1\0\0\0\0\0\0\0", 8)         = 8
nanosleep({tv_sec=0, tv_nsec=10000000}, 0x7ffec64dd3a0) = 0
write(8, "\1\0\0\0\0\0\0\0", 8)         = 8
futex(0x7ffec64dd354, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
close(6)                                = 0
close(7)                                = 0
close(8)                                = 0
close(3)                                = 0
munmap(0x7f7e9bd6d000, 69632)           = 0
munmap(0x7f7e9bd7e000, 1314816)         = 0
munmap(0x7f7e9bc6a000, 1060864)         = 0
munmap(0x7f7e9bedb000, 8192)            = 0
munmap(0x7f7e9beda000, 4096)            = 0
write(10, "\0\0\0\0\377\377\377\377", 8) = 8
futex(0x7f7e9bb00b70, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
munmap(0x7f7e9b9fe000, 1060864)         = 0
epoll_ctl(11, EPOLL_CTL_DEL, 9, 0x7ffec64dd2ec) = 0
close(11)                               = 0
munmap(0x7f7e9bb01000, 28672)           = 0
munmap(0x7f7e9bb08000, 86016)           = 0
munmap(0x7f7e9bb3d000, 172032)          = 0
munmap(0x7f7e9bb28000, 86016)           = 0
munmap(0x7f7e9bb1d000, 45056)           = 0
futex(0x7f7e9bc69964, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f7e9bf27ed4, FUTEX_WAKE_PRIVATE, 1) = 1
munmap(0x7f7e9bb67000, 1060864)         = 0
madvise(0x7f7e9bec2000, 4096, MADV_FREE) = 0
madvise(0x7f7e9bec0000, 4096, MADV_FREE) = 0
munmap(0x7f7e9bebf000, 28672)           = 0
munmap(0x7f7e9bec6000, 36864)           = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT TERM], 8) = 0
rt_sigprocmask(SIG_SETMASK, [HUP INT TERM], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT TERM], 8) = 0
rt_sigprocmask(SIG_SETMASK, [HUP INT TERM], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT TERM], 8) = 0
rt_sigprocmask(SIG_SETMASK, [HUP INT TERM], NULL, 8) = 0
munmap(0x7f7e9beeb000, 4096)            = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [HUP INT TERM], 8) = 0
rt_sigprocmask(SIG_SETMASK, [HUP INT TERM], NULL, 8) = 0
munmap(0x7f7e9bee7000, 4096)            = 0
munmap(0x7f7e9beea000, 4096)            = 0
munmap(0x7f7e9bee9000, 4096)            = 0
munmap(0x7f7e9bee8000, 4096)            = 0
munmap(0x7f7e9bee1000, 8192)            = 0
munmap(0x7f7e9bee3000, 8192)            = 0
munmap(0x7f7e9bed8000, 8192)            = 0
munmap(0x7f7e9bed4000, 16384)           = 0
munmap(0x7f7e9bee5000, 8192)            = 0
munmap(0x7f7e9becf000, 20480)           = 0
close(4)                                = 0
close(5)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

Oct 07 '22 14:10 keisari-ch

I was able to resolve the issue. Thought of sharing it here, not sure if it helps.

Changing our forward to upstream servers vs /etc/resolv.conf helped resolve the issue. i.e. in Corefile forward . 168.244.20.13 168.244.21.13. << works forward . "/etc/resolv.conf" <<errors, alpine resolution

Oct 07 '22 22:10 kranthirachakonda

Hi there.

Still trying to get all the details around a similar issue.

We are using Kubernetes clusters on Azure. (coredns 1.8.7) And our base images are Alpine images. We tested various 3.15 and 3.16 images.

DNS resolution often fails on alpine based pods. We tried debian and ubuntu based images within the same clusters (and same coredns). and name resolution is fine.

Some posts suggests this is due to the lack of TCP support in the MUSL library (https://gitlab.alpinelinux.org/alpine/tsc/-/issues/43)

Behaviour :
/ # cat /etc/resolv.conf
search <any-namespace>.svc.cluster.local svc.cluster.local cluster.local 
nameserver 10.132.0.10   # kube-dns svc ip
options ndots:5
/ # curl https://gitlab.com
curl: (6) Could not resolve host: gitlab.com
/ # curl https://gitlab.com.
<html><body>You are being <a href="https://about.gitlab.com/">redirected</a>.</body></html>/ #

Can you share your Corefile content:

Oct 07 '22 22:10 kranthirachakonda

That is the default i guess for every aks cluster with 1.8.7

apiVersion: v1
data:
  Corefile: |
    aks-test.hcp.switzerlandnorth.azmk8s.io {
        hosts privatelink.hosts aks-test.hcp.switzerlandnorth.azmk8s.io {
          10.0.0.4 aks-test.hcp.switzerlandnorth.azmk8s.io
          fallthrough
        }
    }
    .:53 {
        errors
        ready
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
        import custom/*.override
    }
    import custom/*.server

Oct 10 '22 06:10 keisari-ch

Yes, looks default corefile. For me changing forward plugin as below helped in fixing the errors. Not sure if that would make any difference. Here 168.244.20.13, 21.13 were our NS defined on worker node.

e.g.

forward . 168.244.20.13 168.244.21.13 << works forward . /etc/resolv.conf <<errors, alpine resolution

Oct 12 '22 13:10 kranthirachakonda

docker-alpine docker-alpine copied to clipboard

coredns 1.8.3 not able to resolve records which has CNAME/ALIAS on alpine 3.9.6, 3.12, 3.16.1 - curl/wget

docker-alpine
docker-alpine copied to clipboard