apisix-ingress-controller icon indicating copy to clipboard operation
apisix-ingress-controller copied to clipboard

Published 20 hours ago • verified publisher icon apache

request help: "404 Route Not Found" in APISIX Ingress Controller

Open rastiehaiev opened this issue 2 years ago • 5 comments

Issue description

% minikube version
minikube version: v1.31.2
commit: fd7ecd9c4599bef9f04c0986c4a0187f98a4396e

I am trying to setup APISIX Ingress controller locally, following the instructions from the docs.

I executed the following commands:

helm install apisix apisix/apisix \
  --set gateway.type=NodePort \
  --set ingress-controller.enabled=true \
  --create-namespace \
  --namespace ingress-apisix \
  --set ingress-controller.config.apisix.serviceNamespace=ingress-apisix \
  --set ingress-controller.config.apisix.adminAPIVersion=v3

kubectl -n ingress-apisix run httpbin --image kennethreitz/httpbin --port 80
kubectl -n ingress-apisix expose pod httpbin --port 80
kubectl -n ingress-apisix apply -f httpbin-sample.yaml

File httpbin-sample.yaml looks like this:

---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: httpserver-route
spec:
  http:
    - name: rule1
      match:
        hosts:
          - local.httpbin.org
        paths:
          - /*
      backends:
        - serviceName: httpbin
          servicePort: 80

I can verify ApisixRoute has been created:

% kubectl -n ingress-apisix describe ar httpserver-route
Name:         httpserver-route
Namespace:    ingress-apisix
Labels:       <none>
Annotations:  <none>
API Version:  apisix.apache.org/v2
Kind:         ApisixRoute
Metadata:
  Creation Timestamp:  2023-10-13T16:08:50Z
  Generation:          1
  Resource Version:    256859
  Self Link:           /apis/apisix.apache.org/v2/namespaces/ingress-apisix/apisixroutes/httpserver-route
  UID:                 fb31ed33-c3c5-4a84-abe1-6b2f8feb861e
Spec:
  Http:
    Backends:
      Service Name:  httpbin
      Service Port:  80
    Match:
      Hosts:
        local.httpbin.org
      Paths:
        /*
    Name:  rule1
Events:    <none>

I obtained the host of APISIX Gateway as described in this doc:

% minikube service apisix-gateway --url -n ingress-apisix
http://<somehost>:30777

However, when I try to connect to APISIX Gateway and execute the following request, I receive 404:

curl -i --location 'http://<somehost>:30777/get?foo1=bar1&foo2=bar2' \
--header 'Host: local.httpbin.org'
HTTP/1.1 404 Not Found
Date: Fri, 13 Oct 2023 16:37:05 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Server: APISIX/3.6.0

{"error_msg":"404 Route Not Found"}

There are no errors in APISIX Ingress Controller or APISIX Gateway pod's logs and no hint that APISIX Ingress Controller at least tried to create route. Could you please help? Is there something I forgot?

Environment

  • your apisix-ingress-controller version (output of apisix-ingress-controller version --long): Image apache/apisix-ingress-controller:1.6.0

  • your Kubernetes cluster version (output of kubectl version): Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

  • if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a): Darwin ROMAN.local 23.0.0 Darwin Kernel Version 23.0.0: Fri Sep 15 14:42:42 PDT 2023; root:xnu-10002.1.13~1/RELEASE_X86_64 x86_64

rastiehaiev avatar Oct 13 '23 16:10 rastiehaiev

APISIX Ingress Controller logs:

% kubectl -n ingress-apisix logs -f apisix-ingress-controller-7c6b5f75cd-wnlrp
2023-10-14T00:26:56+08:00	info	ingress/ingress.go:128	apisix ingress controller started
2023-10-14T00:26:56+08:00	info	ingress/ingress.go:130	version:
Version: 1.6.0
Git SHA: no-git-module
Go Version: go1.19.4
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64

2023-10-14T00:26:56+08:00	info	ingress/ingress.go:140	use configuration
{
  "cert_file": "/etc/webhook/certs/cert.pem",
  "key_file": "/etc/webhook/certs/key.pem",
  "log_level": "info",
  "log_output": "stderr",
  "log_rotate_output_path": "",
  "log_rotation_max_size": 100,
  "log_rotation_max_age": 0,
  "log_rotation_max_backups": 0,
  "http_listen": ":8080",
  "https_listen": ":8443",
  "ingress_publish_service": "",
  "ingress_status_address": [],
  "enable_profiling": true,
  "kubernetes": {
    "kubeconfig": "",
    "resync_interval": "6h0m0s",
    "namespace_selector": [],
    "election_id": "ingress-apisix-leader",
    "ingress_class": "apisix",
    "ingress_version": "networking/v1",
    "watch_endpoint_slices": false,
    "api_version": "apisix.apache.org/v2",
    "enable_gateway_api": false
  },
  "apisix": {
    "admin_api_version": "v3",
    "default_cluster_name": "default",
    "default_cluster_base_url": "http://apisix-admin.ingress-apisix.svc.cluster.local:9180/apisix/admin",
    "default_cluster_admin_key": "******"
  },
  "apisix-resource-sync-interval": "1h0m0s",
  "plugin_metadata_cm": ""
}
2023-10-14T00:26:56+08:00	warn	api/server.go:76	failed to load x509 key pair, will not start admission server	{"Error": "open /etc/webhook/certs/cert.pem: no such file or directory", "CertFilePath": "/etc/webhook/certs/cert.pem", "KeyFilePath": "/etc/webhook/certs/key.pem"}
I1014 00:26:56.172984       1 leaderelection.go:248] attempting to acquire leader lease ingress-apisix/ingress-apisix-leader...
2023-10-14T00:26:56+08:00	warn	providers/controller.go:175	found a new leader apisix-ingress-controller-7c6b5f75cd-vw6wh
2023-10-14T00:26:56+08:00	info	providers/controller.go:177	controller now is running as a candidate	{"namespace": "ingress-apisix", "pod": "apisix-ingress-controller-7c6b5f75cd-wnlrp"}
2023-10-14T00:26:57+08:00	info	[email protected]/context.go:173	path: /healthz, status: 200, method: GET, query: , ip: 172.17.0.1, user-agent: kube-probe/1.17, errors: , cost: 819.664µs
2023-10-14T00:27:04+08:00	info	[email protected]/context.go:173	path: /healthz, status: 200, method: GET, query: , ip: 172.17.0.1, user-agent: kube-probe/1.17, errors: , cost: 38.161µs
2023-10-14T00:27:07+08:00	info	[email protected]/context.go:173	path: /healthz, status: 200, method: GET, query: , ip: 172.17.0.1, user-agent: kube-probe/1.17, errors: , cost: 139.565µs
2023-10-14T00:27:13+08:00	info	providers/controller.go:137	LeaderElection	{"message": "apisix-ingress-controller-7c6b5f75cd-wnlrp became leader", "event_type": "Normal"}
I1014 00:27:13.621975       1 leaderelection.go:258] successfully acquired lease ingress-apisix/ingress-apisix-leader
2023-10-14T00:27:13+08:00	warn	providers/controller.go:175	found a new leader apisix-ingress-controller-7c6b5f75cd-wnlrp
2023-10-14T00:27:13+08:00	info	providers/controller.go:359	controller tries to leading ...	{"namespace": "ingress-apisix", "pod": "apisix-ingress-controller-7c6b5f75cd-wnlrp"}
2023-10-14T00:27:13+08:00	info	apisix/stream_route.go:38	resource stream_routes is disabled
2023-10-14T00:27:13+08:00	info	apisix/cluster.go:172	syncing cache	{"cluster": "default"}
2023-10-14T00:27:13+08:00	warn	apisix/cluster.go:344	waiting cluster default to ready, it may takes a while
2023-10-14T00:27:13+08:00	info	apisix/cluster.go:384	syncing schema	{"cluster": "default"}
2023-10-14T00:27:13+08:00	info	apisix/cluster.go:176	cache synced	{"cost_time": "69.534772ms", "cluster": "default"}
2023-10-14T00:27:13+08:00	warn	apisix/cluster.go:355	cluster default now is ready, cost time 70.186096ms
2023-10-14T00:27:13+08:00	info	namespace/namespace_provider.go:120	label selector watching namespaces	{"namespaces": ["default", "ingress-apisix", "kube-node-lease", "kube-public", "kube-system", "roman"]}
W1014 00:27:13.788853       1 reflector.go:424] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Ingress: the server could not find the requested resource
E1014 00:27:13.788964       1 reflector.go:140] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Ingress: failed to list *v1.Ingress: the server could not find the requested resource
2023-10-14T00:27:14+08:00	info	[email protected]/context.go:173	path: /healthz, status: 200, method: GET, query: , ip: 172.17.0.1, user-agent: kube-probe/1.17, errors: , cost: 51.455µs
W1014 00:27:14.825255       1 reflector.go:424] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Ingress: the server could not find the requested resource
E1014 00:27:14.825340       1 reflector.go:140] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Ingress: failed to list *v1.Ingress: the server could not find the requested resource
2023-10-14T00:27:17+08:00	info	[email protected]/context.go:173	path: /healthz, status: 200, method: GET, query: , ip: 172.17.0.1, user-agent: kube-probe/1.17, errors: , cost: 38.068µs
W1014 00:27:17.515709       1 reflector.go:424] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Ingress: the server could not find the requested resource
E1014 00:27:17.515938       1 reflector.go:140] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Ingress: failed to list *v1.Ingress: the server could not find the requested resource
W1014 00:27:21.419923       1 reflector.go:424] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Ingress: the server could not find the requested resource
E1014 00:27:21.421280       1 reflector.go:140] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Ingress: failed to list *v1.Ingress: the server could not find the requested resource
2023-10-14T00:27:24+08:00	info	[email protected]/context.go:173	path: /healthz, status: 200, method: GET, query: , ip: 172.17.0.1, user-agent: kube-probe/1.17, errors: , cost: 57.755µs
2023-10-14T00:27:27+08:00	info	[email protected]/context.go:173	path: /healthz, status: 200, method: GET, query: , ip: 172.17.0.1, user-agent: kube-probe/1.17, errors: , cost: 36.273µs
W1014 00:27:29.127081       1 reflector.go:424] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: failed to list *v1.Ingress: the server could not find the requested resource
E1014 00:27:29.127286       1 reflector.go:140] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:169: Failed to watch *v1.Ingress: failed to list *v1.Ingress: the server could not find the requested resource

rastiehaiev avatar Oct 13 '23 16:10 rastiehaiev

Ran ingress controller with config.logLevel=debug and I can see that whenever I create ApisixRoute, the Ingress controller prints this in log:

2023-10-16T16:51:07+08:00	debug	apisix/apisix_route.go:546	ApisixRoute add event arrived	{"key": "ingress-apisix/httpserver-route", "object": {"metadata":{"name":"httpserver-route","namespace":"ingress-apisix","selfLink":"/apis/apisix.apache.org/v2/namespaces/ingress-apisix/apisixroutes/httpserver-route","uid":"c4960d6e-40bc-467b-8e8d-7ca07733b5ce","resourceVersion":"273447","generation":1,"creationTimestamp":"2023-10-16T08:51:07Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apisix.apache.org/v2\",\"kind\":\"ApisixRoute\",\"metadata\":{\"annotations\":{},\"name\":\"httpserver-route\",\"namespace\":\"ingress-apisix\"},\"spec\":{\"http\":[{\"backends\":[{\"serviceName\":\"httpbin\",\"servicePort\":80}],\"match\":{\"hosts\":[\"local.httpbin.org\"],\"paths\":[\"/get\"]},\"name\":\"rule1\"},{\"backends\":[{\"serviceName\":\"httpbin\",\"servicePort\":80}],\"match\":{\"hosts\":[\"local.httpbin.org\"],\"paths\":[\"/headers\"]},\"name\":\"rule2\"}]}}\n"}},"spec":{"http":[{"name":"rule1","match":{"paths":["/get"],"hosts":["local.httpbin.org"]},"backends":[{"serviceName":"httpbin","servicePort":80,"weight":null}],"websocket":false,"authentication":{"enable":false,"type":"","keyAuth":{},"jwtAuth":{}}},{"name":"rule2","match":{"paths":["/headers"],"hosts":["local.httpbin.org"]},"backends":[{"serviceName":"httpbin","servicePort":80,"weight":null}],"websocket":false,"authentication":{"enable":false,"type":"","keyAuth":{},"jwtAuth":{}}}]},"status":{}}}

However, no route is being created in APISIX:

curl --location 'http://127.0.0.1:9180/apisix/admin/routes'

{"total":0,"list":[]}

Also, in ETCD under the /apisix/routes key there are no records created.

rastiehaiev avatar Oct 16 '23 08:10 rastiehaiev

Hi @rastiehaiev , have u managed to resolve this? I'm encountering the same issue atm.

dchien234 avatar Oct 21 '23 01:10 dchien234

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Mar 14 '24 01:03 github-actions[bot]

@dchien234 I just updated the version of minikube and it worked.

rastiehaiev avatar Mar 14 '24 09:03 rastiehaiev

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Jun 13 '24 01:06 github-actions[bot]

Looks like this was resolved so I'm closing the issue.

For reference, I have recently run into this issue as well when using deploying apisix Ingress controller on a kind cluster on macOS. The solution is to wait for a few more seconds. However when deploying apisix Ingress controller on a kind cluster on ubuntu, it was ok. I do not know why this was the case.

kayx23 avatar Jun 17 '24 01:06 kayx23