respond with servfail if dnssec validation fails

[buffrr]

security-aware resolvers like hsd should protect users from forged DNSSEC responses. hsd only omits the "AD" flag from the response but it should return SERVFAIL instead.

example:

dig @1.1.1.1 dnssec-failed.org a

this will return SERVFAIL with 1.1.1.1 but hsd will return an A record

[chjj]

Agreed. I originally wanted stub resolvers to check the AD bit, but we should probably just return SERVFAIL.

Will get this in for the next release.

[buffrr]

the problem with checking the AD bit only by non validating resolvers is that it's not possible to determine whether the AD bit is missing because the domain is not dnssec signed or whether it was a security failure (bogus response)

google.com doesn't support dnssec so the AD bit is always missing but dig @1.1.1.1 google.com a still returns an A record.

libunbound helps differentiate these cases by having secure and bogus in ub_result. It explains in the comments that the result can be !secure and !bogus at the same time https://github.com/NLnetLabs/unbound/blob/master/libunbound/unbound.h#L186

this makes it easy because if bogus is true then the response should be servfail to prevent tampering attacks for secure domains while also not breaking domains that don't use dnssec.