envoy
envoy copied to clipboard
envoyproxy
ext_authz(http): Read headers sent by ExtAuthz service in the next lua filter
We are using ExtAuthz http service, call to ExtAuthz service fails sometime either due to connection issue(call never reaches ExtAuthz service) or due to some problem in the ExtAuthz processing the request.
In ExtAuthz failure case, I want to retry the request which ExtAuthz static configuration doesn't support yet. so I added a Lua filter after ExtAuthz filter which explicitly calls the ExtAuthz service.
In the next Lua filter I am trying to read the headers set by ExtAuthz service but is not readable. When ExtAuthz service returns 5XX response, next lua filter cannot access the headers set by ExtAuthz service. When ExtAuthz service returns 200 response, next lua filter can access the headers set by ExtAuthz service.
Is there any way to get the ExtAuthz service headers in next lua filter for failure case?
In below example, I want to read some-status header coming from ExtAuthz service.
Envoy configuration
admin:
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: ${ENVOY_ADMIN_PORTNUMBER}
static_resources:
listeners:
- name: listener_0
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: ${ENVOY_PORTNUMBER}
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
http_protocol_options:
accept_http_10: true
stat_prefix: ingress_http
stream_idle_timeout: 900s
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
log_format:
json_format:
"start_time": "%START_TIME%"
"request_id": "%REQ(X-REQUEST-ID)%"
http_filters:
- name: envoy.filters.http.ext_authz
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
clear_route_cache: true
transport_api_version: V3
include_peer_certificate: true
failure_mode_allow: true
failure_mode_allow_header_add: true
http_service:
server_uri:
uri: localhost:8080
cluster: ext_authz-http-service
timeout: 100s
path_prefix: /auth
authorization_response:
dynamic_metadata_from_headers:
patterns:
- exact: "some-status"
- exact: "authorization"
allowed_upstream_headers:
patterns:
- exact: "authorization"
- exact: "x-correlationid"
- exact: "some-status"
authorization_request:
allowed_headers:
patterns:
- exact: "authorization"
- exact: "Authorization"
- exact: "host"
- exact: "x-request-id"
- exact: "x-correlationid"
- name: envoy.filters.http.lua
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
default_source_code:
filename: "/etc/extauth_retries.lua"
source_codes:
some.lua:
inline_string: |
function envoy_on_request(request_handle)
request_handle:headers():add("X-Forwarded-Host", request_handle:headers():get("host"))
local path_without_querystring = string.match(request_handle:headers():get(":path"), "([^?]+)")
request_handle:headers():add("X-Forwarded-Path", path_without_querystring)
end
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
route_config:
name: local_route
request_headers_to_remove:
- "some-status"
virtual_hosts:
- name: some_api_domains
domains: ["api.*"]
retry_policy:
retry_on: "gateway-error,reset,connect-failure,refused-stream,http3-post-connect-failure:"
num_retries: 1
routes:
- match:
safe_regex:
google_re2: { }
regex: "^/someapi/[^/]+$"
headers:
- name: ":method"
safe_regex_match:
google_re2: { }
regex: "PATCH"
route:
regex_rewrite:
pattern:
google_re2: { }
regex: "/someapi"
substitution: "/someprefix"
timeout: 300s
cluster: some-cluster
auto_host_rewrite: true
append_x_forwarded_host: true
clusters:
- name: some-cluster
type: LOGICAL_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: some-cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: someurl.com
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
sni: someurl.com
- name: ext_authz-http-service
connect_timeout: 5s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: ext_authz-http-service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: localhost
port_value: 8080
Lua config
function envoy_on_request(request_handle)
request_handle:logInfo("--------------LUA CALLED--------------")
local some_status = request_handle:headers():get("some-status")
-- local metadata_key1 = request_handle:streamInfo():dynamicMetadata():get("envoy.filters.http.ext_authz")
-- request_handle:logInfo("envoy.filters.http.ext_authz::::::::: " .. tostring(metadata_key1))
request_handle:logInfo("-------EXTAUTHZ HEADERS-------")
for key, value in pairs(request_handle:headers()) do
request_handle:logInfo("-------HEADER1-------"..key)
request_handle:logInfo("-------HEADER2-------"..value)
end
}
Output When EXTAUTHZ returns 200
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:303] [Tags: "ConnectionId":"1","StreamId":"15812020538326292789"] ext_authz filter added header(s) to the request:
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:305] [Tags: "ConnectionId":"1","StreamId":"15812020538326292789"] 'some-status':'200'
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:333] [Tags: "ConnectionId":"1","StreamId":"15812020538326292789"] ext_authz filter removed header(s) from the request:
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][trace][http] [source/common/http/filter_manager.cc:68] [Tags: "ConnectionId":"1","StreamId":"15812020538326292789"] continuing filter chain: filter=0x27ad3f8337c0
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][trace][lua] [./source/extensions/filters/common/lua/lua.h:149] creating N5Envoy10Extensions11HttpFilters3Lua19StreamHandleWrapperE at 0x691e2d1a5310
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: --------------LUA CALLED--------------
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][trace][lua] [./source/extensions/filters/common/lua/lua.h:149] creating N5Envoy10Extensions11HttpFilters3Lua16HeaderMapWrapperE at 0x691e2d1a54c0
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][trace][lua] [./source/extensions/filters/common/lua/lua.h:149] creating N5Envoy10Extensions11HttpFilters3Lua17StreamInfoWrapperE at 0x691e2d1a55f0
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][trace][lua] [./source/extensions/filters/common/lua/lua.h:149] creating N5Envoy10Extensions11HttpFilters3Lua25DynamicMetadataMapWrapperE at 0x691e2d1a5678
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: envoy.filters.http.ext_authz::::::::: table: 0x691e2d1a56b8
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------EXTAUTHZ HEADERS-------
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][trace][lua] [./source/extensions/filters/common/lua/lua.h:149] creating N5Envoy10Extensions11HttpFilters3Lua17HeaderMapIteratorE at 0x691e2d1a58e8
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------:authority
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------envoy:10000
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------:path
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------/someapi
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------:method
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------GET
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------:scheme
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------http
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------accept
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------application/json, text/plain, */*
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------user-agent
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------axios/1.6.8
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------accept-encoding
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------gzip, compress, deflate, br
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------x-forwarded-proto
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------http
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------x-request-id
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------75f88c2c-d3ee-4217-a566-98550b9e411e
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------some-status
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------200
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][trace][lua] [./source/extensions/filters/common/lua/lua.h:210] marking dead N5Envoy10Extensions11HttpFilters3Lua17HeaderMapIteratorE at 0x691e2d1a58e8
[36menvoy_proxy |[0m [2024-04-09 20:23:59.832][17][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: --------------LUA ENDED--------------
Output When EXTAUTHZ returns 500
[36menvoy_proxy |[0m [2024-04-09 20:20:48.628][18][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:455] [Tags: "ConnectionId":"1","StreamId":"9347929444149948615"] ext_authz filter allowed the request with error
--------------LUA CALLED--------------
[36menvoy_proxy |[0m [2024-04-09 20:20:48.628][18][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:455] [Tags: "ConnectionId":"1","StreamId":"9347929444149948615"] ext_authz filter allowed the request with error
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][trace][http] [source/common/http/filter_manager.cc:68] [Tags: "ConnectionId":"1","StreamId":"9347929444149948615"] continuing filter chain: filter=0x17003f8337c0
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][trace][lua] [./source/extensions/filters/common/lua/lua.h:149] creating N5Envoy10Extensions11HttpFilters3Lua19StreamHandleWrapperE at 0x78c242cb6300
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: --------------LUA CALLED--------------
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][trace][lua] [./source/extensions/filters/common/lua/lua.h:149] creating N5Envoy10Extensions11HttpFilters3Lua16HeaderMapWrapperE at 0x78c242cb64b0
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][trace][lua] [./source/extensions/filters/common/lua/lua.h:149] creating N5Envoy10Extensions11HttpFilters3Lua17StreamInfoWrapperE at 0x78c242cb65e0
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][trace][lua] [./source/extensions/filters/common/lua/lua.h:149] creating N5Envoy10Extensions11HttpFilters3Lua25DynamicMetadataMapWrapperE at 0x78c242cb6668
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: envoy.filters.http.ext_authz::::::::: nil
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------EXTAUTHZ HEADERS-------
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][trace][lua] [./source/extensions/filters/common/lua/lua.h:149] creating N5Envoy10Extensions11HttpFilters3Lua17HeaderMapIteratorE at 0x78c242cb6770
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------:authority
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------envoy:10000
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------:path
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------/someapi
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------:method
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------GET
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------:scheme
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------http
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------accept
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------application/json, text/plain, */*
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------authorization
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------Bearer your_auth_token
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------user-agent
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------axios/1.6.8
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------accept-encoding
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------gzip, compress, deflate, br
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------x-forwarded-proto
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------http
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------x-request-id
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------2a57fbaa-fa99-456b-affd-a75be79c2151
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER1-------x-envoy-auth-failure-mode-allowed
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: -------HEADER2-------true
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][trace][lua] [./source/extensions/filters/common/lua/lua.h:210] marking dead N5Envoy10Extensions11HttpFilters3Lua17HeaderMapIteratorE at 0x78c242cb6770
[36menvoy_proxy |[0m [2024-04-09 20:20:48.629][18][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:920] script log: --------------EXECUTING LUA RETRY FOR--------------x_request_id_header 2a57fbaa-fa99-456b-affd-a75be79c2151
When ExtAuthz service returns 5XX response, next lua filter cannot access the headers set by ExtAuthz service.
Is the request rejected at this point? If so, the filter after ext_authz won't run.
Hello @htuch
I have failure_mode_allow: true which allows ExtAuthz filter to pass the request to next Lua filter for 5XX but I want to identify (read status code) the cause of ExtAuthz failure in next Lua filter which I am unable to do as all the headers seems to be removed by ExtAuthz filter when it returns 5XX.
Mainly I want to identify 2 failures, 500 and 503. our ExtAuthz server returns 500 if there is any failure in processing the request. Also, sometime Envoy doesn't able to reach ExtAuthz server and fails with 503 (reported here with logs https://github.com/envoyproxy/envoy/issues/32058).
I want to retry for 503 case (connection issue) in the next Lua filter but don't want to retry for 500 case (processing error by ExtAuthz server).
How can i identify the status code of ExtAuthz filter in the next Lua filter or any other way I can achieve the same?
Reading https://github.com/envoyproxy/envoy/blob/741da8ba26daf0dac5d3e7f02055a494d54a74f0/source/extensions/filters/http/ext_authz/ext_authz.cc#L448, this just takes the failure_mdoe_allowed logic and doesn't apply any headers (as done above in https://github.com/envoyproxy/envoy/blob/741da8ba26daf0dac5d3e7f02055a494d54a74f0/source/extensions/filters/http/ext_authz/ext_authz.cc#L290), which makes sense given 5xx semantics.
One option might be to enhance failure_mode_allow_header_add to support status propagation.
This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.