upstream not found warning in logs

[DGolubets]

Issue description

I get lots of logs, like these:

2024-04-04T22:48:23+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "1a9104d9", "url": "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/1a9104d9", "cluster": "default"}
2024-04-04T22:48:24+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "1a9104d9", "url": "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/1a9104d9", "cluster": "default"}
2024-04-04T23:18:11+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "1a9104d9", "url": "http://apisix-admin.apisix.svc.cluster.local:9180/apisix/admin/upstreams/1a9104d9", "cluster": "default"}

I'm trying to understand why.

I don't have any upstream resources in k8s, but I have routes. Those routes have created 2 upstreams which I can see in the dashboard, but they have different IDs to what I see in the logs.

So my question narrows down to: why does the ingress controller keep looking for the upstream "1a9104d9" and where does it get this id number from?

Environment

• your apisix-ingress-controller version (output of apisix-ingress-controller version --long):

Version: 1.7.1
Git SHA: no-git-module
Go Version: go1.20.12
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64

• your Kubernetes cluster version (output of kubectl version):

Server Version: v1.26.9

[rkazak07]

I'm getting the same warnings too

[rkazak07]

Hello, When I examine the debug mode warnings. I saw that the problem was from Rabbitmq.. It was fixed when I created these upstreams for rabbitmq. But I'm not sure how accurate this approach is. Why is it searching for rabbitmq in an apisix with 0 installations inside?

[Revolyssup]

Actually the endpoint controller in apisix ingress controller tries to look up an upstream resource corresponding to each service endpoint that is created. I am looking into this.

[erSitzt]

Hi any news on this @Revolyssup ? I just set up apisix and was wondering what upstreams it is referring to. What is that ID it is logging ? My apisix config is empty as of now, so where does the ID come from or is it generated for all endpoints found in the cluster ? Generating warnings for all endpoints which i might not want to handle in apisix seems a bit too much.

[nitramiz]

Getting this too. Using latest APISIX chart (2.8.1) with ing-controller subchart (0.14.0).

...
2024-08-28T22:39:32+08:00	info	apisix/apisix_cluster_config.go:65	ApisixClusterConfig controller started
2024-08-28T22:39:32+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "b5de7bb3", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/b5de7bb3", "cluster": "default"}
2024-08-28T22:39:32+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "1c84870a", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/1c84870a", "cluster": "default"}
2024-08-28T22:39:32+08:00	warn	apisix/cluster.go:1164	upstream not found	{"id": "79536e36", "url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/79536e36", "cluster": "default"}
...