apisix
apisix copied to clipboard
apache
bug: The route plugin and consumer plugin are run repeatedly. Is this the correct result?
Current State
apisix version: 3.4.1
https://apisix.apache.org/docs/apisix/3.4/terminology/plugin/
While configuring the same plugin, only one copy of the configuration is valid. The order of precedence is always Consumer > Consumer Group > Route > Plugin Config > Service.
see "only one copy of the configuration is valid"
During implementation, it was found that the same plug-in first executed the routing configuration plug-in, and then executed the consumer configuration plug-in. Is this correct?
Desired State
Documentation is consistent with actual behavior
During implementation, it was found that the same plug-in first executed the routing configuration plug-in, and then executed the consumer configuration plug-in. Is this correct?
What plugin is this and steps to reproduce?
I'm using a custom plugin, outputting logs during the rewrite phase for verification.
local core = require("apisix.core")
local _M = {
version = 0.1,
priority = 22000,
name = "wm-test",
schema = {
type = "object",
properties = {},
}
}
function _M.check_schema(conf)
return core.schema.check(schema, conf)
end
function _M.rewrite(conf, ctx)
core.log.error(core.json.encode(conf))
end
return _M
The following one request logs:
The first line "max_body_size":1000 is the plugin configuration set in the route.
The second line "max_body_size":2097152 is the plugin configuration set in the consumer.
2024/02/26 17:38:58 [error] 90#90: *144 [lua] wm-test.lua:48: phase_func(): trace_id: ac11e468b8dd269df4f9bec53394e6be {"_meta":{"disable":false},"max_body_size":1000}, client: 127.0.0.1, server: _, request: "POST /api3/page/save HTTP/1.1", host: "127.0.0.1:9080"
2024/02/26 17:38:58 [error] 90#90: *144 [lua] wm-test.lua:48: phase_func(): trace_id: ac11e468b8dd269df4f9bec53394e6be {"_meta":{"disable":false},"types":{"multipart/form-data":20000000},"max_body_size":2097152}, client: 127.0.0.1, server: _, request: "POST /api3/page/save HTTP/1.1", host: "127.0.0.1:9080"
However, I deleted the wm-test plug-in in the consumer, and found that the wm-test plug-in in the routing was actually executed twice?
run_plugin(): phase= rewrite & rewrite_in_consumer
#route plugin config::
"wm-test":{
"_meta": {
"disable": false
},
"name": "router"
}
After several attempts later, once _meta.priority was specified, it executed just once!
run_plugin(): phase= rewrite
#route plugin config::
"wm-test":{
"_meta": {
"disable": false,
"priority":991
},
"name": "router"
}
Is it necessary to configure priority in the route plugin?
And sometimes there are different results between two workers, as the 49# process is missing a rewrite log.
I found the code location apisix/init.lua, and the following two places will cause the routing plugin to run twice.
Should only one of these two places be executed?
@kayx23 @hanqingwu It seems that this problem has been around for a while. Is there any progress?
-- in the rewrite phase, the plugin executes in the following order:
-- 1. execute the rewrite phase of the plugins on route(including the auth plugins)
-- 2. merge plugins from consumer and route
-- 3. execute the rewrite phase of the plugins on consumer(phase: rewrite_in_consumer)
-- in this case, we need to skip the plugins that was already executed(step 1)
I don't think this is a problem.
-- in the rewrite phase, the plugin executes in the following order: -- 1. execute the rewrite phase of the plugins on route(including the auth plugins) -- 2. merge plugins from consumer and route -- 3. execute the rewrite phase of the plugins on consumer(phase: rewrite_in_consumer) -- in this case, we need to skip the plugins that was already executed(step 1)I don't think this is a problem.我认为这不是问题。
I think this comment does not meet the current problem. The problem now is that when a plug-in is configured on the routing, if other plug-ins are also configured on the consumer, the rewrite method of the routing plug-in will be executed repeatedly. Isn't this a problem?
Example: Using the proxy-rewrite plug-in (add request header "x-test") on the route will result in adding two "x-test", which is obviously a problem.
cc: @shreemaan-abhishek
relevant issue: https://github.com/apache/apisix/issues/10873
my test
curl "http://127.0.0.1:9180/apisix/admin/consumers" -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"username": "jack",
"plugins": {
"serverless-post-function": {
"functions" : ["return function(conf, ctx)
ngx.say(\"serverless-post-function-consumer\");
end"]
},
"key-auth": {
"key": "auth-one"
}
}
}'
curl -i http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/ip",
"name": "abcd",
"plugins": {
"serverless-post-function": {
"functions" : ["return function(conf, ctx)
ngx.say(\"serverless-post-function-route\");
end"]
},
"key-auth": {
}
},
"upstream": {
"nodes": [
{
"host": "httpbin.org",
"port": 80,
"weight": 1
}
],
"type": "roundrobin",
},
"status": 1
}'
curl http://127.0.0.2:9080/ip -H 'apikey: auth-one' -i
serverless-post-function-consumer
Thanks, please see my test, @hanqingwu
###
PUT http://127.0.0.1:9180/apisix/admin/consumers
X-API-KEY: edd1c9f034335f136f87ad84b625c8f1
Content-Type: application/json
{
"username": "jack",
"plugins": {
"key-auth": {
"key": "auth-one"
}
}
}
###
PUT http://127.0.0.1:9180/apisix/admin/global_rules/1
X-API-KEY: edd1c9f034335f136f87ad84b625c8f1
Content-Type: application/json
{
"plugins": {
"key-auth": {
}
}
}
###
PUT http://127.0.0.1:9180/apisix/admin/routes/1
X-API-KEY: edd1c9f034335f136f87ad84b625c8f1
Content-Type: application/json
{
"uri": "/ip",
"name": "abcd",
"plugins": {
"proxy-rewrite": {
"headers": {
"add": {
"x-test": "123"
}
}
},
"serverless-post-function": {
"functions": [
"return function(conf, ctx) \n ngx.log(ngx.ERR,'router') \n ngx.say(ngx.req.get_headers()['x-test']); \n end"
]
}
},
"upstream": {
"nodes": [
{
"host": "httpbin.org",
"port": 80,
"weight": 1
}
],
"type": "roundrobin"
},
"status": 1
}
###
GET http://127.0.0.1:9080/ip
apikey: auth-one
response repeat:
123123
I do reproduce test case
curl "http://127.0.0.1:9180/apisix/admin/consumers" -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"username": "jack",
"plugins": {
"key-auth": {
"key": "auth-one"
}
}
}'
curl -i http://127.0.0.1:9180/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"uri": "/ip",
"name": "abcd",
"plugins": {
"key-auth":{},
"proxy-rewrite": {
"headers": {
"add": {
"xtest": "123"
}
}
},
"serverless-post-function": {
"functions": [
"return function(conf, ctx) \n ngx.log(ngx.ERR,'router') \n ngx.say(ngx.req.get_headers().xtest); \n end"
]
}
},
"upstream": {
"nodes": [
{
"host": "httpbin.org",
"port": 80,
"weight": 1
}
],
"type": "roundrobin"
},
"status": 1
}'
curl http://127.0.0.2:9080/ip -H 'apikey: auth-one' -i 123123
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] radixtree.lua:355: pre_insert_route(): path: /ip operator: =, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] radixtree.lua:355: pre_insert_route(): path: /apisix/plugin/jwt/sign operator: =, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] radixtree_host_uri.lua:161: match(): route match mode: radixtree_host_uri, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] init.lua:632: http_access_phase(): matched route: {"clean_handlers":{},"key":"/apisix/routes/1","createdIndex":21,"modifiedIndex":68,"value":{"create_time":1709178376,"id":"1","plugins":{"serverless-post-function":{"phase":"access","functions":["return function(conf, ctx) \n ngx.log(ngx.ERR,router) \n ngx.say(ngx.req.get_headers().xtest); \n end"]},"key-auth":{"query":"apikey","header":"apikey","hide_credentials":false},"proxy-rewrite":{"headers":{"add":{"xtest":"123"}},"use_real_request_uri_unsafe":false}},"uri":"/ip","name":"abcd","priority":0,"upstream":{"type":"roundrobin","parent":{"clean_handlers":"table: 0x7fb3253bb308","key":"/apisix/routes/1","createdIndex":21,"modifiedIndex":68,"value":"table: 0x7fb325412b18","orig_modifiedIndex":68,"has_domain":true},"nodes":[{"host":"httpbin.org","priority":0,"port":80,"weight":1}],"pass_host":"pass","scheme":"http","hash_on":"vars"},"status":1,"update_time":1709602849},"orig_modifiedIndex":68,"has_domain":true}, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] consumer.lua:64: create_obj_fun(): consumer:{"modifiedIndex":67,"auth_conf":{"key":"auth-one"},"create_time":1709178297,"consumer_name":"jack","id":"jack","plugins":{"key-auth":{"key":"auth-one"}},"username":"jack","update_time":1709602843}, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] consumer.lua:105: create_obj_fun(): consumer node: {"modifiedIndex":67,"auth_conf":{"key":"auth-one"},"create_time":1709178297,"consumer_name":"jack","id":"jack","plugins":{"key-auth":{"key":"auth-one"}},"username":"jack","update_time":1709602843}, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] secret.lua:233: fetch_secrets(): retrieve secrets refs, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] key-auth.lua:93: phase_func(): consumer: {"modifiedIndex":67,"auth_conf":{"key":"auth-one"},"create_time":1709178297,"consumer_name":"jack","id":"jack","plugins":{"key-auth":{"key":"auth-one"}},"username":"jack","update_time":1709602843}, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] key-auth.lua:106: phase_func(): hit key-auth rewrite, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [warn] 71#71: *112480 [lua] json.lua:95: failed to encode: Cannot serialise, excessive nesting (1001) force: nil, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [warn] 71#71: *112480 [lua] json.lua:95: failed to encode: Cannot serialise, excessive nesting (1001) force: nil, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] plugin.lua:726: merge_consumer_route(): route conf: , client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] plugin.lua:727: merge_consumer_route(): consumer conf: {"modifiedIndex":67,"auth_conf":{"key":"auth-one"},"create_time":1709178297,"consumer_name":"jack","id":"jack","plugins":{"key-auth":{"key":"auth-one"}},"username":"jack","update_time":1709602843}, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] plugin.lua:728: merge_consumer_route(): consumer group conf: null, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [warn] 71#71: *112480 [lua] json.lua:95: failed to encode: Cannot serialise, excessive nesting (1001) force: nil, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [warn] 71#71: *112480 [lua] json.lua:95: failed to encode: Cannot serialise, excessive nesting (1001) force: nil, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] plugin.lua:720: create_obj_fun(): merged conf : , client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] init.lua:709: http_access_phase(): find consumer jack, config changed: true, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [error] 71#71: *112480 [lua] [string "return function(conf, ctx) ..."]:2: func(): nil, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] client.lua:123: dns_parse(): dns resolve httpbin.org, result: {"type":1,"class":1,"ttl":30,"section":1,"name":"httpbin.org","address":"174.129.27.151"}, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
2024/03/05 01:42:47 [info] 71#71: *112480 [lua] resolver.lua:84: parse_domain(): parse addr: {"type":1,"class":1,"ttl":30,"section":1,"name":"httpbin.org","address":"174.129.27.151"}, client: 100.168.34.1, server: _, request: "GET /ip HTTP/1.1", host: "127.0.0.2:9080"
when I explicit set proxy-rewrite disable false in consumers, then get correct header
curl "http://127.0.0.1:9180/apisix/admin/consumers" -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"username": "jack",
"plugins": {
"key-auth": {
"key": "auth-one"
},
"proxy-rewrite": {
"_meta": {
"disable": false
}
}
}
}'
curl http://127.0.0.2:9080/ip -H 'apikey: auth-one' -i 123
Thanks, however, this method is too strange. I first modified it according to my case, and now it is consistent with the document (but there is no support for opening the auth plug-in in routing) https://github.com/dbit-xia/apisix/commits/release/3.4/
I think when call merge_consumer_route , if consumer undefined proxy-rewrite, then after merge will get route plugin config proxy-rewrite , so plugin will run proxy-rewrite add header again.
I also encountered the same problem，and in k8s, according to https://github.com/apache/apisix/issues/10945#issuecomment-1978001342, but it did not take effect
