prameshj

Thanks for sharing this. However this does not include the "setup_errors_total" metric. This metric is exposed on 9353 port. The other coredns metrics from prometheus plugin are exposed on 9253....

Creating this issue to identify what is causing the extra cache misses and if any changes in cache plugin in 1.8.3 could explain it. /cc @chrisohaver

The issue can be repro'ed by running a large number od dnsclient pods that are using the image and parameters [here](https://github.com/kubernetes/perf-tests/blob/d151c4b0b65162e82d9faf21d527bbdd204d0287/clusterloader2/testing/load/deployment.yaml#L29) and [here](https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/testing/load/configmap.yaml). In the automated test, roughly 3k pods...

> Is the ratio of pod:node higher in the 3k pod test than smaller scale tests? Yes, the ratio is much higher. In the smaller test, I only ran a...

Thanks for some great leads! Let me try a smaller scale test with similar pod density and also try 1.8.4 coreDNS and report back.

I haven't had a chance to try it in 1.8.5 yet. I tried to update dependency but kept running into errors like: ``` /go/pkg/mod/k8s.io/[email protected]/logs/config.go:41:22: cannot use nil as type logr.Logger...

@chrisohaver , any thoughts on the prev comment? Thanks!

> I compared the 1.8.2 and 1.8.3 tags to confirm there were no code changes between those two tags ... > > [coredns/[email protected]](https://github.com/coredns/coredns/compare/v1.8.2...v1.8.3) Yup, i noticed the same too. Now,...

I was able to repro in 1.7.1 through 1.8.3 - the tests pass if the number of kube-dns replicas are increased(so there are more DNS pods to serve the cache...

I tried to revert https://github.com/coredns/coredns/pull/4085 on top of coredns 1.7.1 - https://github.com/kubernetes/dns/commit/373ebff7354b7c264c934313f933f6a4cb2aa879 But, nodelocaldns cache misses still show the dnssec bit set - Cache misses from coredns 1.7.0 do not...