bug: Apisix keeps throwing "warn apisix/upstream.go:69 upstream not found"

[adriano-brito-ts]

Current Behavior

APISIX Ingress controller keeps throwing these warns:

warn apisix/upstream.go:69 upstream not found {"name": "dev_echo-service_10052_service", "url": "http://apisix-admin.apisix-ingress.svc.cluster.local:9180/apisix/admin/upstreams/f0b15d95", "cluster": "default"}

I created a route and upstream and they seem like this:

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: echo-service
  namespace: dev
spec:
  http:
  - name: default-route
    match:
      hosts:
      - echo.home.lab
      paths:
      - "/echo/*"
    backends:
    - serviceName: echo-service
      servicePort: 10052
    plugins:
    - name: redirect
      enable: true
      config: 
        http_to_https: true
    - name: proxy-rewrite
      enable: true
      config:
        regex_uri: ["^/echo(/|$)(.*)", "/$2"]
---
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  name: echo-service
  namespace: dev
spec:
  loadbalancer:
    type: least_conn
  healthCheck:
    active:
      type: http
      httpPath: "/v1/status"
      timeout: 5
      healthy:
        successes: 3
        interval: 2s
        httpCodes:
          - 200
          - 206

When I delete the resources, it now keeps throwing additional warn messages like these:

apisix-ingress-controller-8dd7c7474-tk62l 2023-08-15T00:22:42+08:00    warn    apisix/upstream.go:69    upstream not found    {"name": "dev_echo-service_10052", "url": "http://apisix-admin.apisix-ingress.svc.cluster.local:9180/apisix/admin/upstreams/7b294759", "cluster": "default"}

apisix-ingress-controller-8dd7c7474-tk62l 2023-08-15T00:22:42+08:00    warn    apisix/upstream.go:69    upstream not found    {"name": "dev_echo-service_10052_service", "url": "http://apisix-admin.apisix-ingress.svc.cluster.local:9180/apisix/admin/upstreams/f0b15d95", "cluster": "default"}

apisix-ingress-controller-8dd7c7474-tk62l 2023-08-15T00:22:42+08:00    warn    apisix/apisix_upstream.go:526    sync ApisixUpstream failed, will retry    {"object": {"Type":3,"Object":{"Key":"dev/echo-service","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":{}}, "error": "apisixupstream.apisix.apache.org \"echo-service\" not found"}

Expected Behavior

I expect these warning not appear as the route and upstream works fine and seems correctly bounded. And it's weird it keeps saying that about already deleted resources. Is there any event delete cache? Is there a way to flush it?

Error Logs

apisix-ingress-controller-8dd7c7474-tk62l 2023-08-15T00:22:42+08:00    warn    apisix/upstream.go:69    upstream not found    {"name": "dev_echo-service_10052", "url": "http://apisix-admin.apisix-ingress.svc.cluster.local:9180/apisix/admin/upstreams/7b294759", "cluster": "default"}

apisix-ingress-controller-8dd7c7474-tk62l 2023-08-15T00:22:42+08:00    warn    apisix/upstream.go:69    upstream not found    {"name": "dev_echo-service_10052_service", "url": "http://apisix-admin.apisix-ingress.svc.cluster.local:9180/apisix/admin/upstreams/f0b15d95", "cluster": "default"}

apisix-ingress-controller-8dd7c7474-tk62l 2023-08-15T00:22:42+08:00    warn    apisix/apisix_upstream.go:526    sync ApisixUpstream failed, will retry    {"object": {"Type":3,"Object":{"Key":"dev/echo-service","OldObject":null,"GroupVersion":"apisix.apache.org/v2"},"OldObject":null,"Tombstone":{}}, "error": "apisixupstream.apisix.apache.org \"echo-service\" not found"}

Steps to Reproduce

1. Installed APISIX with Manifest generated with helm
2. Created route and upstream with ApisixRoute and ApisixUpstream resources
3. Watch apisix ingress controller logs

Environment

• APISIX Ingress controller version (run apisix-ingress-controller version --long)

Version: 1.6.1
Git SHA: no-git-module
Go Version: go1.19.8
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64

• Kubernetes cluster version (run kubectl version)

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.0", GitCommit:"1b4df30b3cdfeaba6024e81e559a6cd09a089d65", GitTreeState:"clean", BuildDate:"2023-04-11T17:10:18Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:51:25Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

• OS version if running APISIX Ingress controller in a bare-metal environment (run uname -a)

N/A

[adriano-brito-ts]

Another really strange behavior is that an upstream keeps appearing from nowhere even though the CRD resource had been deleted. Why is that?

[tao12345666333]

Sorry for the delay. I was in the hospital the last few days and came back today.

In APISIX Ingress, it will generate the upstream ID through specific rules and try to obtain the actual value from APISIX.

This is related to CRD to some extent, but not entirely dependent on it, as we also need to consider Ingress resources that directly use service name and port.

APISIX Ingress obtains this data in order to establish caching and maintain some relationships between resources, making it more efficient. At the same time, this behavior does not have any impact on the performance of APISIX (although some 404 logs do appear).

[ducnm0711]

Another really strange behavior is that an upstream keeps appearing from nowhere even though the CRD resource had been deleted. Why is that?

Same bug happened to me. Deployed apisix ingress controller, no Upstream defined, the logs filled with

apisix/upstream.go:69	upstream not found	{"name": "default_kubelet_10250", "url": "http://apisix-admin:9180/apisix/admin/upstreams/5b3e9143", "cluster": "default"}
types/types.go:212	upstream is not referenced	{"cluster": "name=default; base_url=http://apisix-admin:9180/apisix/admin", "upstream": "default_kubelet_4194"}

[fatpa]

Refer to https://github.com/apache/apisix-ingress-controller/blob/master/pkg/providers/apisix/apisix_upstream.go#L257, apisixUpstreamController.sync() run twice time to update upstream objects via endpoint and service.

But in the production environment, we use one of them in ApisixRouter backend resolveGranularity. I think it should be changed to update objects that are used in the ApisixRouter. This action will lead to too many Kube warning events do appear.

[github-actions[bot]]

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[wgqkirk]

所以到底要不要手动添加apisixupstream,日志里充满了apisix/upstream.go:70 upstream not found

[github-actions[bot]]

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.