open-im-server icon indicating copy to clipboard operation
open-im-server copied to clipboard

resolve the prometheus discovery issue of same service‘s multiple instances

Open deanwong opened this issue 11 months ago • 4 comments

Description When setAutoPorts enabled, if starting two rpc service (e.g. user rpc service) , the second one will overwrite the etcd value with same key, the code in start.go as below

if autoSetPorts {
	listener, prometheusPort, err = getAutoPort()
	if err != nil {
		return err
	}

	etcdClient := client.(*etcd.SvcDiscoveryRegistryImpl).GetClient()
        // this step will overwrtite the same service with different value
	_, err = etcdClient.Put(ctx, prommetrics.BuildDiscoveryKey(rpcRegisterName), jsonutil.StructToJsonString(prommetrics.BuildDefaultTarget(registerIP, prometheusPort)))
	if err != nil {
		return errs.WrapMsg(err, "etcd put err")
	}
}

For example, I started user service, the etcd value like

openim/prometheus_discovery/user {"target":"10.102.22.175:40145","labels":{"namespace":"default"}}

if I started another user service in different node, the etcd value was covered

openim/prometheus_discovery/user {"target":"/168.64.9.42:34657","labels":{"namespace":"default"}}

Fixed Add nodeId and port in etcd key, the value will be like this in same scenario

openim/prometheus_discovery/user/10.102.22.175:40145 {"target":"10.102.22.175:40145","labels":{"namespace":"default"}}
openim/prometheus_discovery/user/168.64.9.42:34657 {"target":"/168.64.9.42:34657","labels":{"namespace":"default"}}

Moreover, the prometheus discovery will return correct node instances,

[
    {
        "targets": [
            "10.102.22.175:40145",
            "168.64.9.42:34657"
        ],
        "labels": {
            "namespace": "default"
        }
    }
]

deanwong avatar Jan 18 '25 11:01 deanwong

🤖 All Contributors have signed the CLA.
The signed information is recorded here
Posted by the CLA Assistant Lite bot.

github-actions[bot] avatar Jan 18 '25 11:01 github-actions[bot]

I have read the CLA Document and I hereby sign the CLA

deanwong avatar Jan 18 '25 11:01 deanwong

I found that the registration to etcd does not consider using a lease, which will be fixed later

deanwong avatar Jan 21 '25 03:01 deanwong

This branch has conflicts that must be resolved

skiffer-git avatar Mar 05 '25 09:03 skiffer-git