bug: resolveGranularity: service fails, when restarted multiple times

[tao12345666333]

I found that resolveGranularity: service no longer takes effect after multiple restarts of the APISIX Ingress controller

xref: #853

[MirtoBusico]

Hi @tao12345666333 the problem appeared after 5 cluster restarts How I've done the tests:

1. start the cluster and wait everything goes green
2. inspect the upstream address with apisix dashboard
3. shutdown the cluster.

I've done this 5 times. In the first four times in the upstream there is the service address; but in the fifth attempt appears the POD address

In apisix dashboard

The kubernetes addresses

Nothing in the apisix log (UTC time -1 hour respect to dashboard):

2022-02-16T12:46:37.271Z | 127.0.0.6 - - [16/Feb/2022:12:46:34 +0000] apisix-admin.apisix.svc.cluster.local:9180 "GET /apisix/admin/upstreams/f3118c88 HTTP/1.1" 404 39 0.002 "-" "Go-http-client/1.1" - - - "http://apisix-admin.apisix.svc.cluster.local:9180"
2022-02-16T12:46:37.271Z | 127.0.0.6 - - [16/Feb/2022:12:46:34 +0000] apisix-admin.apisix.svc.cluster.local:9180 "GET /apisix/admin/upstreams/23ef4d0 HTTP/1.1" 404 39 0.002 "-" "Go-http-client/1.1" - - - "http://apisix-admin.apisix.svc.cluster.local:9180"
2022-02-16T12:46:37.271Z | 127.0.0.6 - - [16/Feb/2022:12:46:34 +0000] apisix-admin.apisix.svc.cluster.local:9180 "GET /apisix/admin/upstreams/4a5806af HTTP/1.1" 404 39 0.002 "-" "Go-http-client/1.1" - - - "http://apisix-admin.apisix.svc.cluster.local:9180"
2022-02-16T12:46:37.271Z | 127.0.0.6 - - [16/Feb/2022:12:46:34 +0000] apisix-admin.apisix.svc.cluster.local:9180 "GET /apisix/admin/upstreams/6b312a6d HTTP/1.1" 404 39 0.003 "-" "Go-http-client/1.1" - - - "http://apisix-admin.apisix.svc.cluster.local:9180"
2022-02-16T12:46:37.271Z | 127.0.0.6 - - [16/Feb/2022:12:46:34 +0000] apisix-admin.apisix.svc.cluster.local:9180 "GET /apisix/admin/upstreams/7dc55665 HTTP/1.1" 404 39 0.002 "-" "Go-http-client/1.1" - - - "http://apisix-admin.apisix.svc.cluster.local:9180"

Nothing in the apisix ingress controller log:

2022-02-16T13:05:35.738Z | [GIN] 2022/02/16 - 21:05:35 | 200 | 38.15µs | 127.0.0.6 | GET "/healthz"
2022-02-16T13:05:39.633Z | [GIN] 2022/02/16 - 21:05:39 | 200 | 27.66µs | 127.0.0.6 | GET "/healthz"
2022-02-16T13:05:45.738Z | [GIN] 2022/02/16 - 21:05:45 | 200 | 38.349µs | 127.0.0.6 | GET "/healthz"
2022-02-16T13:05:49.633Z | [GIN] 2022/02/16 - 21:05:49 | 200 | 39.56µs | 127.0.0.6 | GET "/healthz"

Recap of framework and installation:

1. three virtual machines cluster with K3S

sysop@m01serv:~$ kubectl get nodes -o wide
NAME     STATUS   ROLES                  AGE   VERSION        INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
m01kw1   Ready    <none>                 28d   v1.22.5+k3s1   192.168.102.122   <none>        Ubuntu 20.04.3 LTS   5.4.0-99-generic   containerd://1.5.8-k3s1
m01kw2   Ready    <none>                 28d   v1.22.5+k3s1   192.168.102.123   <none>        Ubuntu 20.04.3 LTS   5.4.0-99-generic   containerd://1.5.8-k3s1
m01km    Ready    control-plane,master   28d   v1.22.5+k3s1   192.168.102.121   <none>        Ubuntu 20.04.3 LTS   5.4.0-99-generic   containerd://1.5.8-k3s1
sysop@m01serv:~$ 

2. Istio version 1.12.2

sysop@m01serv:~$ istioctl version
client version: 1.12.2
control plane version: 1.12.2
data plane version: 1.12.2 (16 proxies)
sysop@m01serv:~$ 

3. Helm chart version

sysop@m01serv:~$ helm show chart apisix/apisix
apiVersion: v2
appVersion: 2.12.0
dependencies:
- condition: etcd.enabled
  name: etcd
  repository: https://charts.bitnami.com/bitnami
  version: 6.2.6
- alias: dashboard
  condition: dashboard.enabled
  name: apisix-dashboard
  repository: https://charts.apiseven.com
  version: 0.4.0
- alias: ingress-controller
  condition: ingress-controller.enabled
  name: apisix-ingress-controller
  repository: https://charts.apiseven.com
  version: 0.9.0
description: A Helm chart for Apache APISIX
icon: https://apache.org/logos/res/apisix/apisix.png
maintainers:
- name: tao12345666333
name: apisix
type: application
version: 0.8.2

sysop@m01serv:~$ 

4. changes in values.yaml

...
gateway:
  type: LoadBalancer
...
  tls:
    enabled: true
    servicePort: 443
    containerPort: 9443
    existingCASecret: "m01cacert"
    certCAFilename: "cert"
    http2:
      enabled: true
...
discovery:
  enabled: true
  registry:
    dns:
        servers:
            - "10.43.0.10:53"
...
dashboard:
  enabled: true


ingress-controller:
  enabled: true

5. Apisix installation:

kubectl create ns apisix
kubectl label namespace apisix istio-injection=enabled
kubectl -n apisix create secret generic m01cacert --from-file=cert=./m01ca.pem
helm install apisix apisix/apisix -f apisix-values.yaml \
--set ingress-controller.config.apisix.serviceNamespace=apisix \
--set ingress-controller.config.apisix.serviceName=apisix-admin \
--set ingress-controller.config.kubernetes.apisixRouteVersion=apisix.apache.org/v2beta3 \
--namespace apisix

6. route definition:

sysop@m01serv:~/software/apisisx$ cat productpage-ar.yaml 
apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
  name: productpage
spec:
  http:
  - name: rule1
    match:
      hosts:
      - www2.m01.net
      paths:
      - /*
    backends:
    - serviceName: productpage
      servicePort: 9080
      resolveGranularity: service

sysop@m01serv:~/software/apisisx$ 

7. route installation:

kubectl -n bookinfo apply -f productpage-ar.yaml

8. Get upstream from apisix pod shell

bash-5.1# curl http://127.0.0.1:9180/apisix/admin/upstreams/49b914ed -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X GET
{"action":"get","count":1,"node":{"value":{"name":"bookinfo_productpage_9080","update_time":1645015594,"labels":{"managed-by":"apisix-ingress-controller"},"nodes":[{"port":9080,"weight":100,"priority":0,"host":"10.42.1.111"}],"id":"49b914ed","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","hash_on":"vars","scheme":"http","type":"roundrobin","pass_host":"pass","create_time":1644952229},"key":"\/apisix\/upstreams\/49b914ed"}}
bash-5.1# 

Hope this helps. Can I inspect anithing else?

[tao12345666333]

@MirtoBusico Thanks! I can reproduce the problem by restarting the APISIX Ingress controller multiple times. I also plan to fix this bug.

[tao12345666333]

#1251 has been merged, we can close this one.

Thanks all @AlinsRan @MirtoBusico