`Failed to query any nsqd` in Kubernetes without error when run as sts

[alexanderturner]

Running NSQ in Kube yields some interesting results when it's split across StatefulSets.

I've set broadcast addresses on lookupd and nsqd however when I try an operation in nsqadmin, it fails with Failed to query any nsqd: in the nsqd logs (without any error). This occurs when I visit the path topics/{topic} and there's no other relevant logging. When proxying the API endpoint over kube, I have no issues accessing services:

curl -X GET http://127.0.0.1:4161/lookup?topic=dispatch
{"channels":[],"producers":[]}

nslookupd

        command:
          - /nsqlookupd
          - -broadcast-address
          - $(HOSTNAME).nsqlookupd.mynamespace.svc.cluster.local

and nsqd

        command:
          - /nsqd
          - -data-path
          - /data
          - -lookupd-tcp-address
          - nsqlookupd-0.nsqlookupd.mynamespace.svc.cluster.local:4160
          - -lookupd-tcp-address
          - nsqlookupd-1.nsqlookupd.mynamespace.svc.cluster.local:4160
          - -lookupd-tcp-address
          - nsqlookupd-2.nsqlookupd.mynamespace.svc.cluster.local:4160
          - -broadcast-address
          - $(HOSTNAME).nsqd.mynamespace.svc.cluster.local

[ploxiln]

I think this is just because there are no nsqd which are known to have that topic. nsqadmin queries nsqlookupd for a topic to find all nsqd which have it (producers), and then nsqadmin queries all those nsqd directly to find message count, depth, paused state, etc etc. I think this is somewhat of a bug in nsqadmin to show this kind of error when there simply are zero nsqd with this topic, but it is a somewhat rare corner case. (In normal operation every topic that exists, exists on some nsqd. But when you pre-create topics and channels using nsqadmin/nsqlookupd to avoid the first-messages-lost-for-non-first-channels issue, this case is possible. But, if you're just getting started with nsqd, just start publishing some messages, topics and channels are created automatically by nsqd.)