unbound icon indicating copy to clipboard operation
unbound copied to clipboard

Published 20 hours ago • verified publisher icon coredns

panic: runtime error: index out of range [0] with length 0

Open Antiarchitect opened this issue 3 years ago • 9 comments

What happened:

Facing panic: runtime error: index out of range [0] with length 0

What you expected to happen:

No runtime panic.

How to reproduce it (as minimally and precisely as possible):

Do not know - happens periodically.

Anything else we need to know?:

Dockerfile:

FROM golang:1.17.6-bullseye AS builder

ARG COREDNS_VERSION=1.9.0
ARG UNBOUND_PLUGIN_VERSION=0.0.7

ARG CGO_ENABLED=1

WORKDIR /coredns

RUN apt-get update \
    && apt-get install -y --no-install-recommends libunbound-dev \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/* \
    && curl -L -s https://github.com/coredns/coredns/archive/refs/tags/v${COREDNS_VERSION}.tar.gz | tar --strip-components=1 -xzf - -C . \
    && go get "github.com/coredns/unbound@v${UNBOUND_PLUGIN_VERSION}" \
    && echo "unbound:github.com/coredns/unbound" >> plugin.cfg \
    && go generate \
    && go build


FROM debian:11.2-slim

RUN apt-get update \
    && apt-get install -y --no-install-recommends ca-certificates libunbound8 \
    && apt-get clean -y \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /coredns/coredns /opt/coredns

CMD ["/opt/coredns"]

Environment:

  • the version of CoreDNS: 1.9.0
  • Corefile:
# Managed by puppet
(common) {
  errors
  log
  prometheus 0.0.0.0:9253
}

(acl_trusted) {
  acl {
    # Some allow net with private and public IPs which I will not reveal
    block
  }
}

. {
    import common
    reload
    cache 3600
    unbound
    debug
}

local. {
    import common
    import acl_trusted
}

xxx.local. {
    import common
    import acl_trusted
    cache 3600
    forward xxx.local. 10.1.1.101 10.1.1.102
}

acme.local. {
    import common
    file zones/acme.local.db {
        reload 10s
    }
}
  • logs, if applicable:
2022-02-10T14:12:17.2839 193.9.249.145 coredns  panic: runtime error: index out of range [0] with length 0                          
2022-02-10T14:12:17.2839 193.9.249.145 coredns
2022-02-10T14:12:17.2839 193.9.249.145 coredns  goroutine 879 [running]:
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin/cache.key({0xc000748258, 0x9b87f4}, 0xc00000e5e8, 0x0)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/cache/cache.go:78 +0x85
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin/cache.(*ResponseWriter).WriteMsg(0xc00016a6c0, 0xc0005cf5f0)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/cache/cache.go:149 +0xd9
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/unbound.(*Unbound).ServeDNS(0xc00014ae10, {0x22c4df8, 0xc000710a20}, {0x22f
6fc8, 0xc00016a6c0}, 0xc0007093b0)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /go/pkg/mod/github.com/coredns/[email protected]/unbound.go:140 +0x538
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin.NextOrFailure({0x1ebf0d6, 0xc000103000}, {0x22a7df8, 0xc0001
4ae10}, {0x22c4df8, 0xc000710a20}, {0x22f6fc8, 0xc00016a6c0}, 0x1e55440)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/plugin.go:80 +0x264
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin/cache.(*Cache).doRefresh(0xc000623ef0, {0x22c4df8, 0xc000710
a20}, {0xc0007093b0, {0x22f7230, 0xc000d0d540}, {0x0, 0x0}, 0x200, 0x0, ...}, ...)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/cache/handler.go:77 +0xd9
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin/cache.(*Cache).ServeDNS(0xc000623ef0, {0x22c4df8, 0xc000710a
20}, {0x22f7230, 0xc000d0d540}, 0xc000079740)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/cache/handler.go:43 +0x90e
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin.NextOrFailure({0x1ebdb91, 0x1}, {0x22a7718, 0xc000623ef0}, {
0x22c4df8, 0xc000710a20}, {0x22f7230, 0xc000d0d540}, 0x4444e5)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/plugin.go:80 +0x264
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin/log.Logger.ServeDNS({{0x22a7718, 0xc000623ef0}, {0xc00064c9f
0, 0x1, 0x1}, {}}, {0x22c4df8, 0xc000710a20}, {0x22f7180, 0xc000d0d500}, ...)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/log/log.go:36 +0x346
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin.NextOrFailure({0x1ec067e, 0xc000748258}, {0x22ad230, 0xc0006
8f2f0}, {0x22c4df8, 0xc000710a20}, {0x22f7180, 0xc000d0d500}, 0x400)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/plugin.go:80 +0x264
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin/errors.(*errorHandler).ServeDNS(0xc00064c900, {0x22c4df8, 0x
c000710a20}, {0x22f7180, 0xc000d0d500}, 0xc000709320)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/errors/errors.go:84 +0x87
2022-02-10T14:12:17.2839 193.9.249.145 coredns  github.com/coredns/coredns/plugin.NextOrFailure({0x1ec6d92, 0x1}, {0x22a77b8, 0xc00064c900}, {
0x22c4df8, 0xc000710a20}, {0x22f7180, 0xc000d0d500}, 0x0)
2022-02-10T14:12:17.2839 193.9.249.145 coredns          /coredns/plugin/plugin.go:80 +0x264
2022-02-10T14:12:17.2842 193.9.249.145 coredns  github.com/coredns/coredns/plugin/metrics.(*Metrics).ServeDNS(0xc000622cf0, {0x22c4df710a20}, {0x22f7338, 0xc00000e5e8}, 0xc000709320)
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /coredns/plugin/metrics/handler.go:27 +0x255
2022-02-10T14:12:17.2842 193.9.249.145 coredns  github.com/coredns/coredns/core/dnsserver.(*Server).ServeDNS(0xc00012ef60, {0x22c4df8, 0xc0007
10a20}, {0x22f7338, 0xc00000e5e8}, 0xc000709320)
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /coredns/core/dnsserver/server.go:280 +0x64d
2022-02-10T14:12:17.2842 193.9.249.145 coredns  github.com/coredns/coredns/core/dnsserver.(*Server).ServePacket.func1({0x22f8e08, 0xc000862a80
}, 0xc000709301)
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /coredns/core/dnsserver/server.go:128 +0x9b
2022-02-10T14:12:17.2842 193.9.249.145 coredns  github.com/miekg/dns.HandlerFunc.ServeDNS(0xc0002dea00, {0x22f8e08, 0xc000862a80}, 0xc00070932
0)
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /go/pkg/mod/github.com/miekg/[email protected]/server.go:37 +0x2f
2022-02-10T14:12:17.2842 193.9.249.145 coredns  github.com/miekg/dns.(*Server).serveDNS(0xc000712360, {0xc0002dea00, 0x27, 0x0}, 0xc000862a80)
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /go/pkg/mod/github.com/miekg/[email protected]/server.go:659 +0x43e
2022-02-10T14:12:17.2842 193.9.249.145 coredns  github.com/miekg/dns.(*Server).serveUDPPacket(0xc000712360, 0x0, {0xc0002dea00, 0x27, 0x200},
{0x22ea3f0, 0xc000010198}, 0xc0003ec6c0, {0x0, 0x0})
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /go/pkg/mod/github.com/miekg/[email protected]/server.go:603 +0x1dc
2022-02-10T14:12:17.2842 193.9.249.145 coredns  created by github.com/miekg/dns.(*Server).serveUDP
2022-02-10T14:12:17.2842 193.9.249.145 coredns          /go/pkg/mod/github.com/miekg/[email protected]/server.go:533 +0x485
  • OS (e.g: cat /etc/os-release):
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Others:

docker version:

Client:
 Version:           18.09.9
 API version:       1.39
 Go version:        go1.11.13
 Git commit:        039a7df9ba
 Built:             Wed Sep  4 16:51:21 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.13
  Git commit:       039a7df
  Built:            Wed Sep  4 16:22:32 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Antiarchitect avatar Feb 10 '22 14:02 Antiarchitect

Per stack trace, unbound is trying to write an invalid response without the question in it. Looks like an unbound problem.

chrisohaver avatar Feb 10 '22 14:02 chrisohaver

https://github.com/coredns/unbound/issues/25 Seems related

Antiarchitect avatar Feb 10 '22 15:02 Antiarchitect

#25 Seems related

It's the same mode of failure. unbound.Unbound.Resolve() is returning a result with an AnswerPacket that does not contain a Question section.

Probably best to figure out why that happens in the first place in github.com/miekg/unbound, but a hacky-fix here would be to add the section if it's missing. E.g. something along the lines of ...

if len(res.AnswerPacket.Question) == 0 {
   res.AnswerPacket.Question = r.Question
}

Although it would be a safer hack to SERVFAIL when len(res.AnswerPacket.Question) == 0. It's likely that in these cases that it's not just the Question that is missing (e.g. default msg struct), and the response is not valid even with the question hacked back in. e.g. lines 121-123:

if err != nil || len(res.AnswerPacket.Question) == 0 { 
 	return dns.RcodeServerFailure, err 
 }

chrisohaver avatar Feb 10 '22 16:02 chrisohaver

Will you make a PR? I'm not good with golang.

Antiarchitect avatar Feb 10 '22 17:02 Antiarchitect

do you have a test case? I think I can paper over it in miekg/unbound, but not sure if that is the best approach

miekg avatar Feb 12 '22 10:02 miekg

I think I can paper over it in miekg/unbound, but not sure if that is the best approach

A papered over solution probably better to live here in the plugin - e.g. to return SERVFAIL if the AnswerPacket is missing a Question.

chrisohaver avatar Feb 23 '22 01:02 chrisohaver

proper fix upstream with https://github.com/miekg/unbound/pull/18. Thanks miekg!

chrisohaver avatar Feb 23 '22 16:02 chrisohaver

Can the fix be applied in this repo since the upstream fix has stalled before being merged? It has been over a year, so I assume it won't be.

Dart2112 avatar Aug 05 '23 05:08 Dart2112

I think my issue might be related to this one since I'm getting same error message "[ERROR] Recovered from panic in server: "dns://:53" runtime error: index out of range [0] with length 0" with unbound plugin when I try to resolve IBM Cloud Kafka broker instance - broker-5-48d5s71514qnmqz3.kafka.svc11.us-south.eventstreams.cloud.ibm.com. Seems like it might be related to trying to resolve long sub-domains, but again not sure.

When I was testing this 9/10 times it throws error message above and only once time it resolves and returns CNAME.

hbzhost avatar Oct 18 '23 14:10 hbzhost

I echo the need for https://github.com/miekg/unbound/pull/18 to be patched in this repo since upstream has stalled on merging in the fix. For others that are looking for a quick not ideal workaround you can manually patch the three line change with the following:

Grab the patch from the PR:

curl -sL -o pr18.patch https://patch-diff.githubusercontent.com/raw/miekg/unbound/pull/18.patch

If you are using a container to build a custom CoreDNS binary with this plugin you can apply it with the following:

RUN go mod download
COPY plugin.cfg $GOPATH/src/github.com/coredns/coredns
RUN make gen
COPY pr18.patch /root/go/pkg/mod/github.com/miekg/unbound\@v0.0.0-20210309082708-dbeefb4cdb29
WORKDIR /root/go/pkg/mod/github.com/miekg/unbound\@v0.0.0-20210309082708-dbeefb4cdb29
RUN patch -p1 < pr18.patch
WORKDIR $GOPATH/src/github.com/coredns/coredns

I have tested this with the latest CoreDNS 1.11.3 pre-release and it works as expected:

$ podman exec coredns drill A 2.0.0.127.zen.spamhaus.org
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 51610
;; flags: qr rd ra ; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; 2.0.0.127.zen.spamhaus.org.	IN	A

;; ANSWER SECTION:
2.0.0.127.zen.spamhaus.org.	60	IN	A	127.0.0.4
2.0.0.127.zen.spamhaus.org.	60	IN	A	127.0.0.2
2.0.0.127.zen.spamhaus.org.	60	IN	A	127.0.0.10

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 392 msec
;; SERVER: 127.0.0.1
;; WHEN: Mon Jun 10 18:09:10 2024
;; MSG SIZE  rcvd: 170

jdoss avatar Jun 10 '24 18:06 jdoss

I clicked a button

miekg avatar Jun 13 '24 15:06 miekg

@miekg you are the best! Thanks dude!

jdoss avatar Jun 13 '24 16:06 jdoss