apisix-ingress-controller
apisix-ingress-controller copied to clipboard
apache
request help: Does apisixroute v2beta3 support multiple plugins?
Issue description
I set an apisxiroute with two plugin:
- forward-auth
- proxy-rewrite
apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
name: route-backend
namespace: default
spec:
http:
- name: srv
match:
paths:
- /api/*
backends:
- serviceName: backend
servicePort: 80
resolveGranularity: service
plugins:
- name: proxy-rewrite
enable: true
config:
regex_uri: ["^/api/(.*)","/$1"]
- name: forward-auth
enable: true
config:
uri: "http://auth.default.svc.cluster.local:1234/auth"
request_headers: ["Authorization"]
upstream_headers: ["user-id"]
client_headers: ["Location"]
It does not work at all, and it would route to an unexpected place.. However, it works when I tried to delete one plugin and keep another one; for example, just keep proxy-rewrite:
apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
name: route-backend
namespace: default
spec:
http:
- name: srv
match:
paths:
- /api/*
backends:
- serviceName: backend
servicePort: 80
resolveGranularity: service
plugins:
- name: proxy-rewrite
enable: true
config:
regex_uri: ["^/api/(.*)","/$1"]
Environment
-
my apisix version 1.13.1-alpine
-
your apisix-ingress-controller version (output of apisix-ingress-controller version --long): 1.4.0
-
your Kubernetes cluster version (output of kubectl version): 1.20.15
-
if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a): No, it's all under official docker image
Obviously, it can configure multiple plugins.
You should be able to query its actual results by calling the Admin API in APISIX
As for your specific question, I think this requires more information
@tao12345666333 Thank you for your reminder, I would show you another components' setting:
- ingress-controller configmap
apiVersion: v1
data:
config.yaml: |
# log options
log_level: "info"
log_output: "stderr"
cert_file: "/etc/webhook/certs/cert.pem"
key_file: "/etc/webhook/certs/key.pem"
http_listen: ":8080"
https_listen: ":8443"
ingress_publish_service:
enable_profiling: true
kubernetes:
kubeconfig: ""
resync_interval: "6h"
app_namespaces:
- "*"
namespace_selector:
- ""
election_id: "ingress-apisix-leader"
ingress_class: "apisix"
ingress_version: "networking/v1"
watch_endpointslices: false
apisix_route_version: "apisix.apache.org/v2beta3"
enable_gateway_api: false
apisix:
default_cluster_base_url: http://apisix-admin.vastaimlbase.svc.cluster.local:9180/apisix/admin
default_cluster_admin_key: "edd1c9f034335f136f87ad84b625c8f1"
default_cluster_name: "default"
kind: ConfigMap
metadata:
name: apisix-configmap
namespace: vastaimlbase
labels:
helm.sh/chart: ingress-controller-0.9.0
app.kubernetes.io/name: ingress-controller
app.kubernetes.io/instance: apisix
app.kubernetes.io/version: "1.4.0"
app.kubernetes.io/managed-by: Helm
- apisix configmap
apiVersion: v1
kind: ConfigMap
metadata:
name: apisix
namespace: vastaimlbase
data:
config.yaml: |-
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
apisix:
node_listen: 9080 # APISIX listening port
enable_heartbeat: true
enable_admin: true
enable_admin_cors: true
enable_debug: false
enable_dev_mode: false # Sets nginx worker_processes to 1 if set to true
enable_reuseport: true # Enable nginx SO_REUSEPORT switch if set to true.
enable_ipv6: true # Enable nginx IPv6 resolver
config_center: etcd # etcd: use etcd to store the config value
# yaml: fetch the config value from local yaml file `/your_path/conf/apisix.yaml`
#proxy_protocol: # Proxy Protocol configuration
# listen_http_port: 9181 # The port with proxy protocol for http, it differs from node_listen and port_admin.
# This port can only receive http request with proxy protocol, but node_listen & port_admin
# can only receive http request. If you enable proxy protocol, you must use this port to
# receive http request with proxy protocol
# listen_https_port: 9182 # The port with proxy protocol for https
# enable_tcp_pp: true # Enable the proxy protocol for tcp proxy, it works for stream_proxy.tcp option
# enable_tcp_pp_to_upstream: true # Enables the proxy protocol to the upstream server
proxy_cache: # Proxy Caching configuration
cache_ttl: 10s # The default caching time if the upstream does not specify the cache time
zones: # The parameters of a cache
- name: disk_cache_one # The name of the cache, administrator can be specify
# which cache to use by name in the admin api
memory_size: 50m # The size of shared memory, it's used to store the cache index
disk_size: 1G # The size of disk, it's used to store the cache data
disk_path: "/tmp/disk_cache_one" # The path to store the cache data
cache_levels: "1:2" # The hierarchy levels of a cache
# - name: disk_cache_two
# memory_size: 50m
# disk_size: 1G
# disk_path: "/tmp/disk_cache_two"
# cache_levels: "1:2"
allow_admin: # http://nginx.org/en/docs/http/ngx_http_access_module.html#allow
- 0.0.0.0/0
- 0.0.0.0/0
# - "::/64"
port_admin: 9180
# Default token when use API to call for Admin API.
# *NOTE*: Highly recommended to modify this value to protect APISIX's Admin API.
# Disabling this configuration item means that the Admin API does not
# require any authentication.
admin_key:
# admin: can everything for configuration data
- name: "admin"
key: edd1c9f034335f136f87ad84b625c8f1
role: admin
# viewer: only can view configuration data
- name: "viewer"
key: 4054f7cf07e344346cd3f287985e76a2
role: viewer
router:
http: 'radixtree_uri' # radixtree_uri: match route by uri(base on radixtree)
# radixtree_host_uri: match route by host + uri(base on radixtree)
ssl: 'radixtree_sni' # radixtree_sni: match route by SNI(base on radixtree)
stream_proxy: # TCP/UDP proxy
only: false
tcp: # TCP proxy port list
- 9100
udp: # UDP proxy port list
- 9200
# dns_resolver:
#
# - 127.0.0.1
#
# - 172.20.0.10
#
# - 114.114.114.114
#
# - 223.5.5.5
#
# - 1.1.1.1
#
# - 8.8.8.8
#
dns_resolver_valid: 30
resolver_timeout: 5
ssl:
enable: false
enable_http2: true
listen_port: 9443
ssl_protocols: "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"
ssl_ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA"
nginx_config: # config for render the template to genarate nginx.conf
error_log: "/dev/stderr"
error_log_level: "warn" # warn,error
worker_rlimit_nofile: 20480 # the number of files a worker process can open, should be larger than worker_connections
event:
worker_connections: 10620
http:
enable_access_log: true
access_log: "/dev/stdout"
access_log_format: "$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\""
access_log_format_escape: default
keepalive_timeout: 60s # timeout during which a keep-alive client connection will stay open on the server side.
client_header_timeout: 60s # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
client_body_timeout: 60s # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
send_timeout: 10s # timeout for transmitting a response to the client.then the connection is closed
underscores_in_headers: "on" # default enables the use of underscores in client request header fields
real_ip_header: "X-Real-IP" # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
real_ip_from: # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
- 127.0.0.1
- 'unix:'
etcd:
host: # it's possible to define multiple etcd hosts addresses of the same etcd cluster.
- "http://apisix-etcd:2379" # multiple etcd address
prefix: "/apisix" # apisix configurations prefix
timeout: 30 # 30 seconds
plugins: # plugin list
- api-breaker
- authz-keycloak
- basic-auth
- batch-requests
- consumer-restriction
- cors
- echo
- fault-injection
- forward-auth
- grpc-transcode
- hmac-auth
- http-logger
- ip-restriction
- ua-restriction
- jwt-auth
- kafka-logger
- key-auth
- limit-conn
- limit-count
- limit-req
- node-status
- openid-connect
- authz-casbin
- prometheus
- proxy-cache
- proxy-mirror
- proxy-rewrite
- redirect
- referer-restriction
- request-id
- request-validation
- response-rewrite
- serverless-post-function
- serverless-pre-function
- sls-logger
- syslog
- tcp-logger
- udp-logger
- uri-blocker
- wolf-rbac
- zipkin
- traffic-split
- gzip
- real-ip
- ext-plugin-pre-req
- ext-plugin-post-req
stream_plugins:
- mqtt-proxy
- ip-restriction
- limit-conn
And I access my service through apisix-gateway
apiVersion: v1
kind: Service
metadata:
name: apisix-gateway
namespace: vastaimlbase
annotations:
labels:
helm.sh/chart: apisix-0.9.2
app.kubernetes.io/name: apisix
app.kubernetes.io/instance: apisix
app.kubernetes.io/version: "2.13.1"
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/service: apisix-gateway
spec:
type: NodePort
externalTrafficPolicy: Cluster
ports:
- name: apisix-gateway
port: 80
targetPort: 9080
nodePort: 31684
protocol: TCP
selector:
app.kubernetes.io/name: apisix
app.kubernetes.io/instance: apisix
all of above YAML actullay generated from helm chart0.9.2; If you consider it still not enough, please let me know. Thank you very much
@mhh12121 When you apply the first YAMl file (two plugins), any error logs reported by the APISIX Ingress Controller?
@tokers No errors at all... just some warnings about other service not referenced; And the apisixroute itself reported 'synced successfully'
@tokers @tao12345666333
Maybe I OMIT something, I also have another apisixroute routing to the same service: backend
I tried to combine them together into single one apisixroute as following:
apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
name: route-backend
namespace: default
spec:
http:
- name: a
match:
paths:
- /api/*
backends:
- serviceName: backend
servicePort: 80
resolveGranularity: service
plugins:
- name: proxy-rewrite
enable: true
config:
regex_uri: ["^/api/(.*)","/$1"]
- name: forward-auth
enable: true
config:
uri: "http://auth.default.svc.cluster.local:1234/auth"
request_headers: ["Authorization"]
upstream_headers: ["user-id"]
client_headers: ["Location"]
- name: b
match:
paths:
- /openapi.json
- /static/*
methods:
- GET
backends:
- serviceName: backend
servicePort: 80
resolveGranularity: service
- name: c
match:
paths:
- /backend_docs
methods:
- GET
backends:
- serviceName: backend
servicePort: 80
resolveGranularity: service
plugins:
- name: proxy-rewrite
enable: true
config:
uri: "/docs"
So to conclude we have three http proxy rules now:
- a
- b
- c
And the result is kinda weird....
Both of the route rules a and b work as expected, but c not work... And I check my service logs, it seems that c also passed through forward-auth !
@tao12345666333 Yes, I did it through
curl "http://100.89.161.142:9180/apisix/admin/routes/" -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1'
BUt I moved my service to another namespace that I got these reponse:
{
"count": 5,
"action": "get",
"node": {
"key": "\/apisix\/routes",
"nodes": [
{
"modifiedIndex": 153,
"value": {
"id": "7ed9320f",
"name": "test_route-backend_c",
"labels": {
"managed-by": "apisix-ingress-controller"
},
"create_time": 1660182821,
"status": 1,
"uris": [
"\/backend_docs"
],
"desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
"methods": [
"GET",
"POST",
"HEAD",
"PATCH"
],
"priority": 0,
"upstream_id": "12a0880b",
"update_time": 1660182821,
"plugin_config_id": "6f589781",
"plugins": {
"proxy-rewrite": {
"uri": "\/docs"
}
}
},
"key": "\/apisix\/routes\/7ed9320f",
"createdIndex": 153
},
{
"modifiedIndex": 156,
"value": {
"id": "90d75323",
"name": "test_route-backend_a",
"labels": {
"managed-by": "apisix-ingress-controller"
},
"create_time": 1660182986,
"status": 1,
"uris": [
"\/api\/*"
],
"desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
"methods": [
"GET",
"POST",
"DELETE",
"PUT",
"HEAD",
"PATCH"
],
"priority": 0,
"upstream_id": "12a0880b",
"update_time": 1660182986,
"plugin_config_id": "6f589781",
"plugins": {
"proxy-rewrite": {
"regex_uri": [
"^\/api\/(.*)",
"\/$1"
]
},
"forward-auth": {
"uri": "http:\/\/auth.test.svc.cluster.local:8084\/user\/v0\/forward_auth",
"ssl_verify": true,
"keepalive_pool": 5,
"request_headers": [
"Authorization"
],
"keepalive_timeout": 60000,
"upstream_headers": [
"user-id"
],
"client_headers": [
"Location"
],
"timeout": 3000,
"keepalive": true
}
}
},
"key": "\/apisix\/routes\/90d75323",
"createdIndex": 156
},
{
"modifiedIndex": 157,
"value": {
"id": "9de0299",
"name": "test_route-backend_b",
"labels": {
"managed-by": "apisix-ingress-controller"
},
"create_time": 1660182986,
"upstream_id": "12a0880b",
"methods": [
"GET"
],
"uris": [
"\/openapi.json",
"\/static\/*"
],
"priority": 0,
"update_time": 1660182986,
"desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
"status": 1
},
"key": "\/apisix\/routes\/9de0299",
"createdIndex": 157
},
{
"modifiedIndex": 19,
"value": {
"id": "f516764f",
"name": "test_route-user_user",
"labels": {
"managed-by": "apisix-ingress-controller"
},
"create_time": 1660115650,
"upstream_id": "aff25bc5",
"uris": [
"\/user\/v0\/login",
"\/user\/v0\/logout",
"\/user\/v0\/refresh_token",
"\/user\/v0\/register",
"\/user\/swagger\/*"
],
"status": 1,
"priority": 0,
"desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
"update_time": 1660115650
},
"key": "\/apisix\/routes\/f516764f",
"createdIndex": 19
},
{
"modifiedIndex": 18,
"value": {
"id": "f8539915",
"name": "test_route-user_rule1",
"labels": {
"managed-by": "apisix-ingress-controller"
},
"create_time": 1660115650,
"uris": [
"\/test",
"\/user\/v0\/users\/*",
"\/user\/v0\/users"
],
"status": 1,
"plugin_config_id": "8daf7973",
"priority": 0,
"upstream_id": "aff25bc5",
"update_time": 1660115650,
"desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
"plugins": {
"forward-auth": {
"uri": "http:\/\/auth.test.svc.cluster.local:8084\/user\/v0\/forward_auth",
"ssl_verify": true,
"keepalive_pool": 5,
"request_headers": [
"Authorization"
],
"keepalive_timeout": 60000,
"upstream_headers": [
"user-id"
],
"client_headers": [
"Location"
],
"timeout": 3000,
"keepalive": true
}
}
},
"key": "\/apisix\/routes\/f8539915",
"createdIndex": 18
}
],
"dir": true
}
}
{
"modifiedIndex": 153,
"value": {
"id": "7ed9320f",
"name": "test_route-backend_c",
"labels": {
"managed-by": "apisix-ingress-controller"
},
"create_time": 1660182821,
"status": 1,
"uris": [
"\/backend_docs"
],
"desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
"methods": [
"GET",
"POST",
"HEAD",
"PATCH"
],
"priority": 0,
"upstream_id": "12a0880b",
"update_time": 1660182821,
"plugin_config_id": "6f589781",
"plugins": {
"proxy-rewrite": {
"uri": "\/docs"
}
}
},
"key": "\/apisix\/routes\/7ed9320f",
"createdIndex": 153
},
so, this is the C route. and in this route, it doesn't include forward-auth plugin
@tao12345666333 Thank you for your response. However, I don't want the route C to use forward-plugin;
From the results of the above Admin API, it does not include the plugin, as you expect.
In the situation you are currently experiencing, the client request is actually failing, right? This requires you to make the request again and check the APISIX logs
@tao12345666333 Thank you. But I did look through the admin's log; With the apisixroute mentioned above, we could see /backend_docs was forced to use forward-auth:
2022/08/12 11:44:50 [warn] 44#44: *3203743 [lua] plugin.lua:750: run_plugin(): forward-auth exits with http status code 400, client: 10.23.4.212, server: _, request: "GET /backend_docs HTTP/1.1", host: "10.23.4.212:31684"
10.23.4.212 - - [12/Aug/2022:11:44:50 +0000] 10.23.4.212:31684 "GET /backend_docs HTTP/1.1" 400 60 0.002 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" - - - "http://10.23.4.212:31684/backend_docs"
@tao12345666333 Eventually I found a tricky solution cuz I noticed above https://github.com/apache/apisix-ingress-controller/issues/1235#issuecomment-1211479761 : The plugin_config_ids of route B and C are the same: "6f589781"
Combined the following log I posted:
@tao12345666333 Thank you. But I did look through the admin's log; With the apisixroute mentioned above, we could see /backend_docs was forced to use forward-auth:
2022/08/12 11:44:50 [warn] 44#44: *3203743 [lua] plugin.lua:750: run_plugin(): forward-auth exits with http status code 400, client: 10.23.4.212, server: _, request: "GET /backend_docs HTTP/1.1", host: "10.23.4.212:31684" 10.23.4.212 - - [12/Aug/2022:11:44:50 +0000] 10.23.4.212:31684 "GET /backend_docs HTTP/1.1" 400 60 0.002 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" - - - "http://10.23.4.212:31684/backend_docs"
I GUESS they mistook two plugins ;
So my solution is TO DIVIDE THOSE INTO TWO SEPARATE PLUGINS BY creating seperate ApisixPluginConfig resources;
for example:
apiVersion: apisix.apache.org/v2beta3
kind: ApisixPluginConfig
metadata:
name: plugin-backend-docs
namespace: default
spec:
plugins:
- name: proxy-rewrite
enable: true
config:
uri: "/docs"
---
apiVersion: apisix.apache.org/v2beta3
kind: ApisixPluginConfig
metadata:
name: plugin-backend-api
namespace: default
spec:
plugins:
- name: forward-auth
enable: true
config:
uri: "http://auth:8084/user/v0/forward_auth"
request_headers: ["Authorization"]
upstream_headers: ["user-id"]
client_headers: ["Location"]
- name: proxy-rewrite
enable: true
config:
regex_uri: ["^/api/(.*)","/$1"]
in apisixRoute, what you need to update is to indicate the plugin_config_name:
apiVersion: apisix.apache.org/v2beta3
kind: ApisixRoute
metadata:
name: route-backend
namespace: default
spec:
http:
- name: a
match:
paths:
- /api/*
backends:
- serviceName: backend
servicePort: 80
resolveGranularity: service
plugin_config_name: plugin-backend-api
- name: b
match:
paths:
- /openapi.json
- /static/*
methods:
- GET
backends:
- serviceName: srv-vastml-backend-amd64-svc
servicePort: 80
resolveGranularity: service
- name: c
match:
paths:
- /backend_docs
plugin_config_name: plugin-backend-docs
backends:
- serviceName: srv-vastml-backend-amd64-svc
servicePort: 80
resolveGranularity: service
And it seems that 1.4.0 ingress-controller would report something like:
"apisixpluginConfig Not registered"
Then I upgraded it to 1.4.1, and it works like a charm!
This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
It looks like the original problem has been solved. I will close it. Thanks