Invalid target after forwarding dns with bind (DNSSEC related)

[tdna]

I went through the tutorial on mesos dns docs, setup bind and the following error came back from mesos dns: ERROR: 2015/06/04 09:09:39 resolver.go:379: invalid target

Direct query to mesos dns works good.

[kozyraki]

@tdna can you provide some more information about your setup and about the specific request. Send us:

• the mesos-dns config file
• the specific request that to bind that led to the error
• the other messages you see on your console if you run with -v=2 (around the request) This will help us understand what the problem is. Thanks

[tdna]

Thanks for your reply!

I used mesos dns docker image. I started with -v=2 mode but nothing around this message only "invalid target" 3 times.

Our mesos dns ip: 192.168.1.115 Our bind server ip is: 192.168.1.147

Bind config:

zone "mesos" {
type forward;
forward only;
forwarders { 192.168.1.115 port 53; };
};

Query to mesos dns

$ dig @192.168.1.115 master.mesos

; <<>> DiG 9.9.5-3-Ubuntu <<>> @192.168.1.115 master.mesos
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52638
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;master.mesos.          IN  A

;; ANSWER SECTION:
master.mesos.       60  IN  A   192.168.1.208

;; Query time: 1 msec
;; SERVER: 192.168.1.115#53(192.168.1.115)
;; WHEN: Fri Jun 05 07:31:18 UTC 2015
;; MSG SIZE  rcvd: 58

Mesos dns log:

ERROR: 2015/06/05 07:33:43 resolver.go:379: invalid target
ERROR: 2015/06/05 07:33:43 resolver.go:379: invalid target
ERROR: 2015/06/05 07:33:43 resolver.go:379: invalid target

Query to our dns server

$ dig master.mesos

; <<>> DiG 9.9.5-3-Ubuntu <<>> master.mesos
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 7349
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;master.mesos.          IN  A

;; Query time: 5 msec
;; SERVER: 192.168.1.147#53(192.168.1.147)
;; WHEN: Fri Jun 05 07:33:43 UTC 2015
;; MSG SIZE  rcvd: 41

Mesos dns log:

ERROR: 2015/06/05 07:33:43 resolver.go:379: invalid target
ERROR: 2015/06/05 07:33:43 resolver.go:379: invalid target
ERROR: 2015/06/05 07:33:43 resolver.go:379: invalid target

Bind server log:

05-Jun-2015 07:33:43.590 error (unexpected RCODE REFUSED) resolving 'master.mesos/DS/IN': 192.168.1.115#53
05-Jun-2015 07:33:43.591 error (no valid DS) resolving 'master.mesos/A/IN': 192.168.1.115#53

Other mesos dns log parts:

VERY VERBOSE: 2015/06/05 07:53:15 generator.go:95: Zookeeper says the leader is:  master-2:5050
VERY VERBOSE: 2015/06/05 07:53:15 generator.go:180: reloading from master master-2
VERY VERBOSE: 2015/06/05 07:53:16 generator.go:184: Warning: master changed to master-2
VERY VERBOSE: 2015/06/05 07:53:16 generator.go:364: [A] mesos-dns.marathon.mesos.: 192.168.1.115
VERY VERBOSE: 2015/06/05 07:53:16 generator.go:364: [A] mesos-dns-17854-s302.marathon.mesos.: 192.168.1.115
VERY VERBOSE: 2015/06/05 07:53:16 generator.go:364: [SRV]   _mesos-dns._tcp.marathon.mesos.: mesos-dns-17854-s302.marathon.mesos.:31806
VERY VERBOSE: 2015/06/05 07:53:16 generator.go:364: [SRV]   _mesos-dns._udp.marathon.mesos.: mesos-dns-17854-s302.marathon.mesos.:31806
...
...
VERY VERBOSE: 2015/06/05 07:56:18 logging.go:63: {MesosRequests:17 MesosSuccess:16 MesosNXDomain:1 MesosFailed:0 NonMesosRequests:8 NonMesosSuccess:0 NonMesosNXDomain:8 NonMesosFailed:0 NonMesosRecursed:0}

Mesos dns config:

{
  "zk": "zk://master-1:2181,master-2:2181,master-3:2181/mesos",
  "masters": ["master-1:5050", "master-2:5050", "master-3:5050"],
  "refreshSeconds": 60,
  "ttl": 60,
  "domain": "mesos",
  "port": 53,
  "resolvers": ["192.168.1.147", "8.8.8.8"],
  "timeout": 5,
  "httpon": true,
  "dsnon": true,
  "httpport": 8123,
  "externalon": true,
  "listener": "0.0.0.0",
  "SOAMname": "root.ns1.mesos",
  "SOARname": "ns1.mesos",
  "SOARefresh": 60,
  "SOARetry":   600,
  "SOAExpire":  86400,
  "SOAMinttl": 60
}

[kozyraki]

@tdna Looking at what you sent me, there are two interesting lines:

05-Jun-2015 07:33:43.590 error (unexpected RCODE REFUSED) resolving 'master.mesos/DS/IN': 192.168.1.115#53
05-Jun-2015 07:33:43.591 error (no valid DS) resolving 'master.mesos/A/IN': 192.168.1.115#53

It looks like you are using DNSSEC and to make this work we need to support DS, DNSKEY, and RRSIG records (see a tutorial here). I will add it to the todo list as this is not a trivial hack.

[tdna]

Yes I am using dnssec. Thanks, it would be useful!

[sepiroth887]

+1 ran into the same issue. It is a bit of a blocker for us as we cannot disable DNSSEC.

[tsenart]

@jdef: How can we prioritise this?

[air]

Hi @tdna and @sepiroth887, if possible can you talk a bit about your organization and why DNSSEC is important (or non-negotiable!) for you? Thanks.

[sepiroth887]

I wrote to hastily. Turns out for us its actually fine to disable dnssec. Not sure how much dnssec buys you anyways considering there are better mechanisms in place to secure dns :D

[tdna]

I've disabled dnssec finally. Now it seems to be working.