Isogram

Ah interesting, querying against the v4 service ip seems to get correctly redirected and proxied. Cilium installed with Helm. I can send the sysdump privately on slack if needed

After running dig @(service ip v4 for kubedns), and greping the client pod's IP: ``` # cilium monitor | grep 10.192.65.231 Listening for events on 96 CPUs with 64x4096 of...

> In the IPv6 case, are you able to confirm that 2001:db8:4001::a is the VIP for the DNS service and 2001:db8:4000:1::7e78 is the IP for the backend DNS pod? Yep,...

> > Cilium installed with Helm. > > > > Please provide the full command. Apologies for the delay, on a phone. https://gist.github.com/gpl/14fcbed7079251fac5b5d797c4d78b25

Attaching profiles for gomaxprocs 1,2,4,8,16. [coredns-gomaxprocs.zip](https://github.com/coredns/coredns/files/9489572/coredns-gomaxprocs.zip)

Hmm, I don't believe either of those are the issue here -- We had previously attempted to adjust a various number of kernel parameters and haven't seen any significant deviance...

Sorry for the lack of clarity - CoreDNS doesn't consume more than 4-5 cores. I rebuilt coredns with symbols and ran `perf` instead of pprof: 

Another weird thing we've noticed - a lot of actions seem to time out at 7 seconds, but it's unclear exactly how that timeout is set -- {"level":"warn","ts":"2022-08-25T01:23:24.880Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2022-08-25T01:23:17.880Z","time...

> @gpl Could you provide detailed steps so that others can try to reproduce it locally? I was able to fix it after poking around in bbolt for a while;...

At least in our use case, we are okay with delays of multiple seconds for data in redis, so we would be ok with replicating off of a redis follower....