bug: ingress-controller always shows this log not found service, but the service is actually there

[GhangZh]

Issue description

ingress-controller always shows this log, but the service is actually there

Environment

• your apisix-ingress-controller version (output of apisix-ingress-controller version --long):1.4.0
• your Kubernetes cluster version (output of kubectl version):1.18
• if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):

Minimal test code / Steps to reproduce

kubectl -n ingress-apisix logs -f apisix-ingress-controller-xxxxxx |grep retry kubectl -n ingress-apisix describe ar apisixroute-dashboard

Actual result

no error log

Error log

2022-02-24T15:39:39+08:00 warn ingress/apisix_route.go:307 sync ApisixRoute failed, will retry {"object": {"Type":1,"Object":{"Key":"ingress-apisix/apisixroute-dashboard","OldObject":null,"GroupVersion":"apisix.apache.org/v2beta1"},"Tombstone":null}, "error": "service "apisix-dashboard" not found"}

Expected result

No response

[tao12345666333]

can you run kubectl api-resources --api-group=apisix.apache.org ?

[GhangZh]

kubectl api-resources --api-group=apisix.apache.org

[tao12345666333]

Are you using the latest CRD? https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix-ingress-controller/crds/customresourcedefinitions.yaml

[GhangZh]

Are you using the latest CRD? https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix-ingress-controller/crds/customresourcedefinitions.yaml

Okay, I'll try it out.

[tao12345666333]

We have made some adjustments to CRD in the new version

[elricli]

Are you using the latest CRD? https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix-ingress-controller/crds/customresourcedefinitions.yaml

Same. It's using ltest CRD.

[tao12345666333]

@drrrMikado can you show me your full logs? And definitions for resources such as ApisixRoute and Service

[elricli]

log:

 2022-03-01T09:49:03+08:00	error	translation/apisix_route.go:338	failed to get service port in backend	{"backend": {"serviceName":"flow-control-server-go-dev","servicePort":80,"weight":null}, "apisix_route": {"kind":"ApisixRoute","apiVersion":"apisix.apache.org/v2beta2","metadata":{"name":"flow-control-server-go-dev","namespace":"flow-control-server-go","selfLink":"/apis/apisix.apache.org/v2beta2/namespaces/flow-control-server-go/apisixroutes/flow-control-server-go-dev","uid":"178a1217-3674-465e-b07b-be7d60acc57c","resourceVersion":"9564966417","generation":1,"creationTimestamp":"2022-02-28T03:51:33Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apisix.apache.org/v2beta3\",\"kind\":\"ApisixRoute\",\"metadata\":{\"annotations\":{},\"name\":\"flow-control-server-go-dev\",\"namespace\":\"flow-control-server-go\"},\"spec\":{\"http\":[{\"backends\":[{\"serviceName\":\"flow-control-server-go-dev\",\"servicePort\":80}],\"match\":{\"hosts\":[\"flow-control-server-go-dev.xxx.com\"],\"paths\":[\"/ws\"]},\"name\":\"websocket\",\"websocket\":true},{\"backends\":[{\"serviceName\":\"flow-control-server-go-dev\",\"servicePort\":80}],\"match\":{\"hosts\":[\"flow-control-server-go-dev.xxx.com\"],\"paths\":[\"/*\"]},\"name\":\"default\"}]}}\n"},"managedFields":[{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2beta2","time":"2022-02-28T03:51:33Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:http":{}},"f:status":{".":{},"f:conditions":{}}}},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-02-28T03:51:33Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{}}}]},"spec":{"http":[{"name":"websocket","match":{"paths":["/ws"],"hosts":["flow-control-server-go-dev.xxx.com"]},"backends":[{"serviceName":"flow-control-server-go-dev","servicePort":80,"weight":null}],"websocket":true,"authentication":{"enable":false,"type":"","keyauth":{}}},{"name":"default","match":{"paths":["/*"],"hosts":["flow-control-server-go-dev.xxx.com"]},"backends":[{"serviceName":"flow-control-server-go-dev","servicePort":80,"weight":null}],"websocket":false,"authentication":{"enable":false,"type":"","keyauth":{}}}]},"status":{"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":1,"lastTransitionTime":null,"reason":"ResourcesSynced","message":"Sync Successfully"}]}}, "error": "service \"flow-control-server-go-dev\" not found"}

 2022-03-01T09:49:03+08:00	error	ingress/apisix_route.go:159	failed to translate ApisixRoute v2beta2	{"error": "service \"flow-control-server-go-dev\" not found", "object": {}}

 2022-03-01T09:49:03+08:00	warn	ingress/apisix_route.go:307	sync ApisixRoute failed, will retry	{"object": {"Type":1,"Object":{"Key":"flow-control-server-go/flow-control-server-go-dev","OldObject":null,"GroupVersion":"apisix.apache.org/v2beta2"},"Tombstone":null}, "error": "service \"flow-control-server-go-dev\" not found"}

// 'platform/account-system' also not found, but log level is info, not error.
 2022-03-01T09:49:03+08:00	info	ingress/controller.go:609	service platform/account-system not found

ApisixRoute:

apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
  name: flow-control-server-go-dev
  namespace: flow-control-server-go
spec:
  http:
  - name: websocket
    match:
      hosts:
      - flow-control-server-go-dev.xxx.com
      paths:
      - /ws
    backends:
      - serviceName: flow-control-server-go-dev
        servicePort: 80
    websocket: true
  - name: default
    match:
      hosts:
      - flow-control-server-go-dev.xxx.com
      paths:
      - /*
    backends:
      - serviceName: flow-control-server-go-dev
        servicePort: 80

Service:

kind: Service
apiVersion: v1
metadata:
  name: flow-control-server-go-dev
  namespace: flow-control-server-go
  labels:
    app: flow-control-server-go-dev
spec:
  ports:
    - name: web
      protocol: TCP
      port: 80
      targetPort: 8000
  selector:
    app: flow-control-server-go-dev
    version: v1
  type: NodePort
  sessionAffinity: None

[tao12345666333]

hi @drrrMikado I see that your ApisixRoute resource is in flow-control-server-go namespace, but Service flow-control-server-go-dev is in flow-control-server-go-dev namespace, so it will not be found.

[elricli]

hi @drrrMikado I see that your ApisixRoute resource is in flow-control-server-go namespace, but Service flow-control-server-go-dev is in flow-control-server-go-dev namespace, so it will not be found.

I'm sorry. It's my fault. I copied it wrong. I updated the above configuration and added screenshots.

[elricli]

Also, I get a different error.

 2022-03-01T17:55:49+08:00	error	ingress/status.go:197	failed to get APISIX gateway external IPs	{"error": "resource name may not be empty"}

 2022-03-01T17:55:49+08:00	error	ingress/status.go:197	failed to get APISIX gateway external IPs	{"error": "resource name may not be empty"}

[tao12345666333]

Could you please post the complete installation and reproduction steps?

Your newly posted log indicates that you have enabled the Ingress LB status function, which requires you to check your Kubernetes environment.

[elricli]

Could you please post the complete installation and reproduction steps?

It took a while for the problem to occur, so I can't give you the reproduction steps.

Your newly posted log indicates that you have enabled the Ingress LB status function, which requires you to check your Kubernetes environment.

Thanks, I will check the Kubernetes environment.

[github-actions[bot]]

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.