There is a performance problem in lb/consul.updater, cpu 100%

[tobeabme]

2.54mins runtime.lock runtime.chanrecv runtime.selectnbrecv code.qschou.com/peduli/go_service_passport/vendor/github.com/olivere/grpc/lb/consul.(*Resolver).updater -----------+------------------------------------------------------- 53.54s runtime.unlock runtime.chanrecv runtime.selectnbrecv code.qschou.com/peduli/go_service_passport/vendor/github.com/olivere/grpc/lb/consul.(*Resolver).updater

profile001.pdf

[tobeabme]

there are the problem code

func (r *Resolver) updater(instances []string, lastIndex uint64) {
	var err error
	var oldInstances = instances
	var newInstances []string

	// TODO Cache the updates for a while, so that we don't overwhelm Consul.
	for {
		select {
		case <-r.quitc:
			break
		default:
			newInstances, lastIndex, err = r.getInstances(lastIndex)
			if err != nil {
				log.Printf("grpc/lb/consul: error retrieving instances from Consul: %v", err)
				continue
			}
			updates := r.makeUpdates(oldInstances, newInstances)
			if len(updates) > 0 {
				r.updatesc <- updates
			}
			oldInstances = newInstances
		}
	}
}

[olivere]

I see. I'm not sure if it's worth fixing the issue as the way gRPC does resolving/balancing is now at a very different level as it used to be at the time I was writing this library. See #1 for a hint. There are alternatives like https://github.com/ekomobile/grpc-consul-resolver that seem to register Consul as a resolver and use the newer LB mechanism.

Looking at this issue, we could of course add a simple cache to updater or use Consul watches to stop polling for changes.