request help: apisix-ingress-controller has some warning logs when it starts

[GhangZh]

Issue description

The apisix-ingress-controller has some warning logs when it starts, please help to make sure it is OK.

1. After upgrading apisix-ingress-controller to 1.4.1, I found that there are more 'due to client-side throttling, not priority and fairness' logs when starting up. This is because of the addition of the limit queue AddRateLimited function? 2.xxx in APISIX but do not in declare yaml Is this log normal?

2022-07-19T15:41:08+08:00	warn	api/server.go:75	failed to load x509 key pair, will not start admission server	{"Error": "open /etc/webhook/certs/cert.pem: no such file or directory", "CertFilePath": "/etc/webhook/certs/cert.pem", "KeyFilePath": "/etc/webhook/certs/key.pem"}
I0719 15:41:08.063070       6 leaderelection.go:248] attempting to acquire leader lease ingress-apisix/ingress-apisix-leader...
2022-07-19T15:41:08+08:00	info	ingress/controller.go:320	LeaderElection	{"message": "apisix-ingress-controller-86b99f4fb7-ksw4v became leader", "event_type": "Normal"}
I0719 15:41:08.075986       6 leaderelection.go:258] successfully acquired lease ingress-apisix/ingress-apisix-leader
2022-07-19T15:41:08+08:00	warn	ingress/controller.go:358	found a new leader apisix-ingress-controller-86b99f4fb7-ksw4v
2022-07-19T15:41:08+08:00	info	ingress/controller.go:406	controller tries to leading ...	{"namespace": "ingress-apisix", "pod": "apisix-ingress-controller-86b99f4fb7-ksw4v"}
2022-07-19T15:41:08+08:00	warn	apisix/cluster.go:332	waiting cluster default to ready, it may takes a while
2022-07-19T15:41:08+08:00	info	apisix/cluster.go:160	syncing cache	{"cluster": "default"}
2022-07-19T15:41:08+08:00	info	apisix/cluster.go:372	syncing schema	{"cluster": "default"}
2022-07-19T15:41:08+08:00	info	apisix/cluster.go:164	cache synced	{"cost_time": "144.015795ms", "cluster": "default"}
2022-07-19T15:41:08+08:00	warn	apisix/cluster.go:343	cluster default now is ready, cost time 144.102063ms
I0719 15:41:09.242007       6 request.go:665] Waited for 1.00035705s due to client-side throttling, not priority and fairness, request: GET:https://xxxxxxxx:443/apis/apisix.apache.org/v2beta3/namespaces/project-data-process/apisixroutes
[GIN] 2022/07/19 - 15:41:10 | 200 |      90.758µs |   192.168.115.1 | GET      "/healthz"
[GIN] 2022/07/19 - 15:41:12 | 200 |      44.735µs |   192.168.115.1 | GET      "/healthz"
I0719 15:41:19.242584       6 request.go:665] Waited for 11.000791185s due to client-side throttling, not priority and fairness, request: GET:https://xxxxxxxx:443/apis/apisix.apache.org/v2beta3/namespaces/project-2080ti-speech/apisixroutes
[GIN] 2022/07/19 - 15:41:20 | 200 |      35.668µs |   192.168.115.1 | GET      "/healthz"
[GIN] 2022/07/19 - 15:41:22 | 200 |      50.562µs |   192.168.115.1 | GET      "/healthz"
I0719 15:41:29.441727       6 request.go:665] Waited for 21.199241829s due to client-side throttling, not priority and fairness, request: GET:https://xxxxxxxx:443/apis/apisix.apache.org/v2beta3/namespaces/fleet-system/apisixroutes
[GIN] 2022/07/19 - 15:41:30 | 200 |      45.716µs |   192.168.115.1 | GET      "/healthz"
[GIN] 2022/07/19 - 15:41:32 | 200 |      41.543µs |   192.168.115.1 | GET      "/healthz"
I0719 15:41:39.442269       6 request.go:665] Waited for 25.597955314s due to client-side throttling, not priority and fairness, request: GET:https://xxxxxxxx:443/apis/apisix.apache.org/v2beta3/namespaces/image-test/apisixtlses
[GIN] 2022/07/19 - 15:41:40 | 200 |      36.259µs |   192.168.115.1 | GET      "/healthz"
[GIN] 2022/07/19 - 15:41:42 | 200 |      34.411µs |   192.168.115.1 | GET      "/healthz"
I0719 15:41:49.641502       6 request.go:665] Waited for 25.597537658s due to client-side throttling, not priority and fairness, request: GET:https://xxxxxxxx:443/apis/apisix.apache.org/v2beta3/namespaces/project-cpu-pilot3/apisixtlses
[GIN] 2022/07/19 - 15:41:50 | 200 |      33.491µs |   192.168.115.1 | GET      "/healthz"
[GIN] 2022/07/19 - 15:41:52 | 200 |      32.894µs |   192.168.115.1 | GET      "/healthz"
I0719 15:41:59.641845       6 request.go:665] Waited for 25.597653718s due to client-side throttling, not priority and fairness, request: GET:https://xxxxxxxx:443/apis/apisix.apache.org/v2beta3/namespaces/team-speech-prod-2080ti/apisixconsumers
[GIN] 2022/07/19 - 15:42:00 | 200 |      41.111µs |   192.168.115.1 | GET      "/healthz"
[GIN] 2022/07/19 - 15:42:02 | 200 |      35.815µs |   192.168.115.1 | GET      "/healthz"
I0719 15:42:09.841566       6 request.go:665] Waited for 25.598399031s due to client-side throttling, not priority and fairness, request: GET:https://xxxxxxxx:443/apis/apisix.apache.org/v2beta3/namespaces/kube-system/apisixconsumers
[GIN] 2022/07/19 - 15:42:10 | 200 |      35.009µs |   192.168.115.1 | GET      "/healthz"
[GIN] 2022/07/19 - 15:42:12 | 200 |      35.335µs |   192.168.115.1 | GET      "/healthz"
I0719 15:42:19.842223       6 request.go:665] Waited for 25.598635335s due to client-side throttling, not priority and fairness, request: GET:https://xxxxxxxx:443/apis/apisix.apache.org/v2beta3/namespaces/aidi-pre-prod/apisixconsumers
[GIN] 2022/07/19 - 15:42:20 | 200 |      33.136µs |   192.168.115.1 | GET      "/healthz"
[GIN] 2022/07/19 - 15:42:22 | 200 |      30.947µs |   192.168.115.1 | GET      "/healthz"
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: 7ffe60f0 in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: c72d086f in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: 52d1131b in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: fadbd928 in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: e33d4d1f in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: 430b4caa in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: ca7b3e0e in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: af513e9e in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: e6f9da80 in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: 95d63dab in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: 6dfa3293 in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: ea1f3daf in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: 1dc8d8b1 in APISIX but do not in declare yaml
2022-07-19T15:42:23+08:00	warn	ingress/compare.go:186	route: 28fa99fc in APISIX but do not in declare 

Environment

• your apisix-ingress-controller version (output of apisix-ingress-controller version --long):1.4.1
• your Kubernetes cluster version (output of kubectl version):1.18
• if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):

[wolgod]

Are there some routes in apisix that are not created through the apisix ingress controller?

[tao12345666333]

These logs are some expected output, but it also means that you are currently facing some situation.

1. Same as 👆 you have some routes not created by APISIX Ingress

2. You are experiencing some current throttling. You can look at older logs, or restart it to see the latest logs

[GhangZh]

These logs are some expected output, but it also means that you are currently facing some situation.

1. Same as 👆 you have some routes not created by APISIX Ingress
2. You are experiencing some current throttling. You can look at older logs, or restart it to see the latest logs

1.Yes, traefik and apisix are currently coexisting, there was a change to the annotaion apisix class of an ingress, but this ingress has now been removed. 2.I've tried restart times and only version 1.4.1 has this problem when starting up

[tao12345666333]

you can enable debug level log to see more details

[github-actions[bot]]

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.