envoyproxy
Dynamic Metadata match not working unless I modify headers via Lua script
Title: Dynamic Metadata match not working unless I modify headers via Lua script
Description: Envoy should be able to match requests to a route based on Dynamic Metadata, and the behavior should not be affected just because I add/remove an arbitrary, unrelated header via Lua script.
In this example, we're decrypting a JWT token and saving the payload fields to dynamic metadata. We then expect the request to match a dynamic metadata matcher if it had brandId field with value of tracfone in the JWT payload. For an unknown reason, it appears this match only works if try to add or remove a header to the request from a Lua script. From our testing, it doesn't matter what the header is, and it doesn't matter where you try to add or remove it. Just modifying the header in some way via Lua does the trick. 🫤
The examples below contain the necessary docker and curl commands (including the JWT header with the expected values) to reproduce the behavior.
Repro steps:
- Save the config listed below to a file
envoy.yaml: - From a terminal in the same folder:
docker run -v ${PWD}/envoy.yaml:/etc/envoy/envoy.yaml -p 10000:10000 envoyproxy/envoy-dev:latest -c /etc/envoy/envoy.yaml
- Once the container is running, run this in a different terminal:
curl -i -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJxdWFsdHJpY3MiLCJhdWRpZW5jZSI6InF1YWx0cmljcyIsImJyYW5kSWQiOiJ0cmFjZm9uZSJ9.bNTKuV1FFwYSJSLc6RmCsAaFwVb8RnXP4doxCmNE3WM" http://localhost:10000/echo-service
Expected Outcome: Request should be matched to the first match entry, and receive a 200 direct response. Actual Outcome: Request us matched to the second match entry, and receives a 403 direct response.
NOTE: If you uncomment the request_handle:headers():remove("A") line in the Lua script and re-try, it will work and you will receive the 200 response code.
Config:
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
http_filters:
- name: envoy.filters.http.jwt_authn
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
providers:
xyz-jwt:
forward: false
payload_in_metadata: jwt_payload
local_jwks:
inline_string: '{
"keys":
[
{
"kty":"oct",
"alg":"HS256",
"use": "sig",
"k": "WFla"
}
]
}'
rules:
- match:
prefix: "/echo-service"
requires:
requires_any:
requirements:
- provider_name: "xyz-jwt"
- allow_missing: {}
- name: envoy.lua
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inline_code: |
function envoy_on_request(request_handle)
-- THIS APPEARS TO BE AN ENVOY BUG. IF WE DON'T HAVE THIS LINE, THE MATCHING ON DYNAMIC METADATA STOPS WORKING.
-- IF UNCOMMENTED: You will get a 200.
-- IF COMMENTED: You will get a 403.
--request_handle:headers():remove("A")
end
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/echo-service"
dynamic_metadata:
filter: envoy.filters.http.jwt_authn
path:
- key: jwt_payload
- key: brandId
value:
string_match:
exact: tracfone
direct_response:
status: 200
body:
inline_string: "Hello world."
- match:
prefix: "/echo-service"
direct_response:
status: 403
body:
inline_string: "Forbidden."
To simplify the repro, here is another config file / request pair that has the same issue. The only difference is that I'm using the header_to_metadata filter to set the dynamic metadata (just to rule out that it had anything to do with the JWT filter). The repro steps are the exact same, just replace the config and use a different request to test it:
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
http_filters:
- name: envoy.filters.http.header_to_metadata
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.header_to_metadata.v3.Config
request_rules:
- header: x-version
on_header_present:
metadata_namespace: envoy.lb
key: version
type: STRING
remove: false
- name: envoy.lua
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inline_code: |
function envoy_on_request(request_handle)
-- THIS APPEARS TO BE AN ENVOY BUG. IF WE DON'T HAVE THIS LINE, THE MATCHING ON DYNAMIC METADATA STOPS WORKING.
-- IF UNCOMMENTED: You will get a 200.
-- IF COMMENTED: You will get a 403.
--request_handle:headers():remove("A")
end
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/echo-service"
dynamic_metadata:
filter: envoy.lb
path:
- key: version
value:
string_match:
exact: "123"
direct_response:
status: 200
body:
inline_string: "Hello world."
- match:
prefix: "/echo-service"
direct_response:
status: 403
body:
inline_string: "Forbidden."
Request:
curl -i -H "x-version: 123" http://localhost:10000/echo-service
This is a non well-documented behavior, Envoy caches the route decision before the request hit filters, and the JWT filter doesn't clear the cache. The Lua filter when it is modifying a header clears the cache as a side effect.
This could be workaround as this, or you can use RBAC filter to reject the request with Forbidden instead of writing them in route config.
I had the same problem. I think the lua filter should clear the cached route when dynamic metadata is modified as same as headers.
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.