envoy
envoy copied to clipboard
envoyproxy
AWS STS API - Expiration field misunderstood - Timestamps are formatted according to the ISO 8601 not unix timestamp
A warning explain than Expiration value is missing or not an integer. I think it's a misunderstood about Expiration field, we can see an example on API documentation. Timestamps are formatted according to the ISO 8601 standard.
You can find the impacted code here, and could be patched quickly with the same logic we can found here
Patch
diff --git a/source/extensions/common/aws/credentials_provider_impl.cc b/source/extensions/common/aws/credentials_provider_impl.cc
index 139d53f177..6cec8e96cd 100644
--- a/source/extensions/common/aws/credentials_provider_impl.cc
+++ b/source/extensions/common/aws/credentials_provider_impl.cc
@@ -688,12 +688,15 @@ void WebIdentityCredentialsProvider::extractCredentials(
setCredentialsToAllThreads(
std::make_unique<Credentials>(access_key_id, secret_access_key, session_token));
- const auto expiration = Utility::getIntegerFromJsonOrDefault(credentials.value(), EXPIRATION, 0);
+ const auto expiration =
+ Utility::getStringFromJsonOrDefault(credentials.value(), EXPIRATION, "");
- if (expiration != 0) {
- expiration_time_ =
- std::chrono::time_point<std::chrono::system_clock>(std::chrono::seconds(expiration));
- ENVOY_LOG(debug, "AWS STS credentials expiration time (unix timestamp): {}", expiration);
+ if (!expiration.empty()) {
+ absl::Time expiration_time;
+ if (absl::ParseTime(EXPIRATION_FORMAT, expiration, &expiration_time, nullptr)) {
+ ENVOY_LOG(debug, "Container role AWS credentials expiration time: {}", expiration);
+ expiration_time_ = absl::ToChronoTime(expiration_time);
+ }
} else {
expiration_time_ = api_.timeSource().systemTime() + REFRESH_INTERVAL;
ENVOY_LOG(warn, "Could not get Expiration value of AWS credentials document from STS, so "
Repro steps:
- EKS 1.28
- ArgoCD
- AWS Controllers for Kubernetes (ACK) - IAM
- AWS Controllers for Kubernetes (ACK) - Lambda
Manifest
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
argocd.argoproj.io/sync-wave: "-3"
eks.amazonaws.com/audience: sts.amazonaws.com
eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/k8s-dev-app
eks.amazonaws.com/sts-regional-endpoints: "true"
eks.amazonaws.com/token-expiration: "43200"
labels:
app.kubernetes.io/instance: envoy
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: envoy
app.kubernetes.io/version: 1.30.1
helm.sh/chart: envoy-1.10.0
name: app
namespace: default
---
apiVersion: iam.services.k8s.aws/v1alpha1
kind: Role
metadata:
annotations:
argocd.argoproj.io/sync-options: PrunePropagationPolicy=background
argocd.argoproj.io/sync-wave: "-3"
labels:
app.kubernetes.io/part-of: ack
name: envoy-invoke-lambda
spec:
assumeRolePolicyDocument: |
{
"Statement": [
{
"Action": "sts:AssumeRole",
"Condition": {
"Bool": {
"aws:MultiFactorAuthPresent": "true"
}
},
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::791324568:root"
},
"Sid": "root"
},
{
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.eu-central-1.amazonaws.com/id/456789123abcdef:sub": "system:serviceaccount:default:app"
}
},
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/456789123abcdef"
},
"Sid": "AssumeRoleWithWebIdentity"
}
],
"Version": "2012-10-17"
}
description: envoy-invoke-lambda
maxSessionDuration: 43200
name: k8s-dev-app
permissionsBoundary: arn:aws:iam::123456789:policy/ack/boundary@dev
policies:
- arn:aws:iam::123456789:policy/k8s-dev-app
tags:
- key: AppName
value: app
- key: CostCenter
value: IT
- key: EnvironmentType
value: DEV
- key: ProjectName
value: project
- key: StackSource
value: https://github.com/envoyproxy/envoy
- key: TeamContact
value: [email protected]
---
apiVersion: iam.services.k8s.aws/v1alpha1
kind: Role
metadata:
annotations:
argocd.argoproj.io/sync-options: PrunePropagationPolicy=background
argocd.argoproj.io/sync-wave: "-3"
labels:
app.kubernetes.io/part-of: ack
name: hello-world
spec:
assumeRolePolicyDocument: |
{
"Statement": [
{
"Action": "sts:AssumeRole",
"Condition": {
"Bool": {
"aws:MultiFactorAuthPresent": "true"
}
},
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::791324568:root"
},
"Sid": "root"
},
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
}
}
],
"Version": "2012-10-17"
}
description: hello-world
maxSessionDuration: 3600
name: k8s-hello-world
permissionsBoundary: arn:aws:iam::123456789:policy/ack/boundary@dev
policies:
- arn:aws:iam::123456789:policy/k8s-hello-world
tags:
- key: AppName
value: app
- key: CostCenter
value: IT
- key: EnvironmentType
value: DEV
- key: ProjectName
value: project
- key: StackSource
value: https://github.com/envoyproxy/envoy
- key: TeamContact
value: [email protected]
---
apiVersion: v1
data:
envoy.yaml: |-
admin:
access_log_path: /dev/stdout
address:
socket_address:
address: 0.0.0.0
port_value: 9901
layered_runtime:
layers:
- name: static_layer
static_layer:
envoy.reloadable_features.use_http_client_to_fetch_aws_credentials: true
static_resources:
listeners:
- name: envoy-http-listener
address:
socket_address:
address: 0.0.0.0
port_value: 8080
protocol: TCP
filter_chains:
- name: httphost-shared
filter_chain_match: {}
filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.file
typed_config:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/fd/1
log_format:
text_format_source:
inline_string: |
ACCESS [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"
http_filters:
- name: envoy.filters.http.aws_lambda
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.aws_lambda.v3.Config
arn: "arn:aws:lambda:eu-central-1:*:function:*"
payload_passthrough: false
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
suppress_envoy_headers: true
normalize_path: true
use_remote_address: true
server_name: envoy
http_protocol_options:
accept_http_10: false
preserve_external_request_id: false
route_config:
name: local_route
virtual_hosts:
- name: lambda
domains: ["*"]
routes:
- match:
prefix: /hello-world/
route:
cluster: lambda
typed_per_filter_config:
envoy.filters.http.aws_lambda:
'@type': type.googleapis.com/envoy.extensions.filters.http.aws_lambda.v3.PerRouteConfig
invoke_config:
arn: "arn:aws:lambda:eu-central-1:123456789:function:k8s-hello-world"
payload_passthrough: false
traffic_direction: UNSPECIFIED
clusters:
- name: lambda
connect_timeout: 3s
type: LOGICAL_DNS
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: lambda
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address: { address: lambda.eu-central-1.amazonaws.com, port_value: 443, protocol: TCP }
metadata:
filter_metadata:
com.amazonaws.lambda:
egress_gateway: true
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
sni: "*.amazonaws.com"
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/instance: envoy
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: envoy
app.kubernetes.io/version: 1.30.1
helm.sh/chart: envoy-1.10.0
name: app
---
apiVersion: v1
data: null
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/instance: envoy
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: envoy
app.kubernetes.io/version: 1.30.1
helm.sh/chart: envoy-1.10.0
name: app-xds
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/instance: envoy
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: envoy
app.kubernetes.io/version: 1.30.1
helm.sh/chart: envoy-1.10.0
name: app
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
selector:
app.kubernetes.io/instance: envoy
app.kubernetes.io/name: envoy
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
argocd.argoproj.io/sync-wave: "0"
labels:
app.kubernetes.io/instance: envoy
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: envoy
app.kubernetes.io/version: 1.30.1
helm.sh/chart: envoy-1.10.0
name: app
spec:
replicas: 3
selector:
matchLabels:
app.kubernetes.io/instance: envoy
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: envoy
app.kubernetes.io/version: 1.30.1
helm.sh/chart: envoy-1.10.0
strategy:
rollingUpdate:
maxSurge: 2
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
annotations:
argocd.argoproj.io/sync-wave: "-3"
checksum/config: a61c9bb99ad3b4688ee6510fab20bf88f3166192adac4105ec3ad63b9ad6a73a
eks.amazonaws.com/audience: sts.amazonaws.com
eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/k8s-dev-app
eks.amazonaws.com/sts-regional-endpoints: "true"
eks.amazonaws.com/token-expiration: "43200"
labels:
app.kubernetes.io/instance: envoy
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: envoy
app.kubernetes.io/version: 1.30.1
helm.sh/chart: envoy-1.10.0
spec:
containers:
- args:
- --log-level
- info
- --config-path
- /config/envoy.yaml
command:
- /docker-entrypoint.sh
env: null
image: envoyproxy/envoy:v1.30.1
imagePullPolicy: IfNotPresent
lifecycle: {}
livenessProbe:
initialDelaySeconds: 30
tcpSocket:
port: admin
name: envoy
ports:
- containerPort: 9901
name: admin
protocol: TCP
- containerPort: 8080
name: http
protocol: TCP
readinessProbe:
initialDelaySeconds: 30
tcpSocket:
port: admin
resources:
limits:
cpu: 500m
memory: 1024Mi
requests:
cpu: 100m
memory: 256Mi
volumeMounts:
- mountPath: /config
name: config
initContainers:
- command:
- sh
- -c
- |
/bin/sh <<'EOF'
set -e
export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \
$(aws sts assume-role-with-web-identity \
--role-arn $AWS_ROLE_ARN \
--role-session-name ${POD_NAME} \
--web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE \
--duration-seconds 900 \
--query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
--output text))
aws lambda invoke --region eu-central-1 --function-name k8s-hello-world /dev/stdout
EOF
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: public.ecr.aws/aws-cli/aws-cli:2.15.41
imagePullPolicy: IfNotPresent
name: aws-cli
securityContext: {}
serviceAccountName: app
terminationGracePeriodSeconds: 30
topologySpreadConstraints:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: envoy
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
- labelSelector:
matchLabels:
app.kubernetes.io/instance: envoy
maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
volumes:
- configMap:
name: app
name: config
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
labels:
app.kubernetes.io/instance: envoy
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: envoy
app.kubernetes.io/version: 1.30.1
helm.sh/chart: envoy-1.10.0
name: app
spec:
maxUnavailable: 1
selector:
matchLabels:
app.kubernetes.io/instance: envoy
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: envoy
app.kubernetes.io/version: 1.30.1
helm.sh/chart: envoy-1.10.0
---
apiVersion: iam.services.k8s.aws/v1alpha1
kind: Policy
metadata:
annotations:
argocd.argoproj.io/sync-options: PrunePropagationPolicy=background
argocd.argoproj.io/sync-wave: "-3"
labels:
app.kubernetes.io/part-of: ack
name: envoy-invoke-lambda
spec:
description: envoy-invoke-lambda
name: k8s-dev-app
path: /
policyDocument: |
{
"Statement": [
{
"Action": [
"lambda:InvokeFunction"
],
"Effect": "Allow",
"Resource": [
"arn:aws:lambda:eu-central-1:123456789:function:k8s-hello-world"
],
"Sid": "invoke"
}
],
"Version": "2012-10-17"
}
tags:
- key: AppName
value: app
- key: CostCenter
value: IT
- key: EnvironmentType
value: DEV
- key: ProjectName
value: project
- key: StackSource
value: https://github.com/envoyproxy/envoy
- key: TeamContact
value: [email protected]
---
apiVersion: iam.services.k8s.aws/v1alpha1
kind: Policy
metadata:
annotations:
argocd.argoproj.io/sync-options: PrunePropagationPolicy=background
argocd.argoproj.io/sync-wave: "-3"
labels:
app.kubernetes.io/part-of: ack
name: hello-world
spec:
description: hello-world
name: k8s-hello-world
path: /
policyDocument: |
{
"Statement": [
{
"Action": "logs:CreateLogGroup",
"Effect": "Allow",
"Resource": "arn:aws:logs:eu-central-1:123456789:*"
},
{
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Effect": "Allow",
"Resource": [
"arn:aws:logs:eu-central-1:123456789:log-group:/aws/lambda/k8s-hello-world:*"
]
}
],
"Version": "2012-10-17"
}
tags:
- key: AppName
value: app
- key: CostCenter
value: IT
- key: EnvironmentType
value: DEV
- key: ProjectName
value: project
- key: StackSource
value: https://github.com/envoyproxy/envoy
- key: TeamContact
value: [email protected]
---
apiVersion: lambda.services.k8s.aws/v1alpha1
kind: Function
metadata:
annotations:
argocd.argoproj.io/sync-options: PruneLast=false,PrunePropagationPolicy=background
argocd.argoproj.io/sync-wave: "-3"
services.k8s.aws/region: eu-central-1
name: hello-world
spec:
architectures:
- x86_64
code:
imageURI: 123456789.dkr.ecr.eu-central-1.amazonaws.com/lambda-hello-world:202311022305
description: function created by ACK lambda-controller
environment:
variables:
LOGGING_LEVEL: INFO
ephemeralStorage:
size: 512
memorySize: 128
name: k8s-hello-world
packageType: Image
publish: true
role: arn:aws:iam::123456789:role/k8s-hello-world
tags:
- key: AppName
value: app
- key: CostCenter
value: IT
- key: EnvironmentType
value: DEV
- key: ProjectName
value: project
- key: StackSource
value: https://github.com/envoyproxy/envoy
- key: TeamContact
value: [email protected]
timeout: 300
Logs:
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:428] initializing epoch 0 (base id=0, hot restart version=11.104)
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:430] statically linked extensions:
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.connection_handler: envoy.connection_handler.default
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.resolvers: envoy.ip
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.network.application_protocols, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.matching.input_matchers: envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip, envoy.matching.matchers.runtime_fraction
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.http.header_validators: envoy.http.header_validators.envoy_default
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.http.injected_credentials: envoy.http.injected_credentials.generic
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] quic.http_server_connection: quic.http_server_connection.default
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.dynatrace
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.health_checkers: envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.dubbo_proxy.protocols: dubbo
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.route.early_data_policy: envoy.route.early_data_policy.default
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.load_balancing_policies: envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.route_config_update_requester: envoy.route_config_update_requester.default
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] network.connection.client: default, envoy_internal
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.geoip_providers: envoy.geoip_providers.maxmind
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.request_id: envoy.request_id.uuid
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.regex_engines: envoy.regex_engines.google_re2
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.formatter: envoy.formatter.cel, envoy.formatter.metadata, envoy.formatter.req_without_query
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.http_capsule
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.dubbo_proxy.serializers: dubbo.hessian2
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.thrift_proxy.transports: auto, framed, header, unframed
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.health_check.event_sinks: envoy.health_check.event_sink.file
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.common.key_value: envoy.key_value.file_based
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.quic.server_preferred_address: quic.server_preferred_address.fixed
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.string_matcher: envoy.string_matcher.lua
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432] envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment
[2024-04-26 02:51:18.031][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.bootstrap.v3.Admin Using deprecated option 'envoy.config.bootstrap.v3.Admin.access_log_path' from file bootstrap.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2024-04-26 02:51:18.031][1][info][main] [source/server/server.cc:486] HTTP header map info:
[2024-04-26 02:51:18.032][1][info][main] [source/server/server.cc:489] request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-ot-span-context,x-request-id
[2024-04-26 02:51:18.032][1][info][main] [source/server/server.cc:489] request trailer map: 120 bytes:
[2024-04-26 02:51:18.032][1][info][main] [source/server/server.cc:489] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2024-04-26 02:51:18.032][1][info][main] [source/server/server.cc:489] response trailer map: 144 bytes: grpc-message,grpc-status
[2024-04-26 02:51:18.037][1][info][main] [source/server/server.cc:861] runtime: layers:
- name: static_layer
static_layer:
envoy.reloadable_features.use_http_client_to_fetch_aws_credentials: true
[2024-04-26 02:51:18.037][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.access_loggers.file.v3.FileAccessLog Using deprecated option 'envoy.extensions.access_loggers.file.v3.FileAccessLog.format' from file file.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2024-04-26 02:51:18.037][1][info][admin] [source/server/admin/admin.cc:66] admin address: 0.0.0.0:9901
[2024-04-26 02:51:18.038][1][info][config] [source/server/configuration_impl.cc:168] loading tracing configuration
[2024-04-26 02:51:18.038][1][info][config] [source/server/configuration_impl.cc:124] loading 0 static secret(s)
[2024-04-26 02:51:18.038][1][info][config] [source/server/configuration_impl.cc:130] loading 1 cluster(s)
[2024-04-26 02:51:18.038][1][info][config] [source/server/configuration_impl.cc:138] loading 1 listener(s)
[2024-04-26 02:51:18.040][1][info][config] [source/server/configuration_impl.cc:154] loading stats configuration
[2024-04-26 02:51:18.040][1][warning][main] [source/server/server.cc:928] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.downstream_connections` resource monitor.
[2024-04-26 02:51:18.040][1][info][main] [source/server/server.cc:969] starting main dispatch loop
[2024-04-26 02:51:18.040][1][info][misc] [source/extensions/common/aws/utility.cc:381] Added a LOGICAL_DNS internal cluster [name: sts_token_service_internal, address:sts.eu-central-1.amazonaws.com:443] to fetch aws credentials
[2024-04-26 02:51:18.041][1][info][misc] [source/extensions/common/aws/utility.cc:381] Added a STATIC internal cluster [name: ec2_instance_metadata_server_internal, address:169.254.169.254:80] to fetch aws credentials
[2024-04-26 02:51:18.041][1][info][misc] [source/extensions/common/aws/utility.cc:381] Added a LOGICAL_DNS internal cluster [name: sts_token_service_internal, address:sts.eu-central-1.amazonaws.com:443] to fetch aws credentials
[2024-04-26 02:51:18.042][1][info][runtime] [source/common/runtime/runtime_impl.cc:614] RTDS has finished initialization
[2024-04-26 02:51:18.042][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:240] cm init: all clusters initialized
[2024-04-26 02:51:18.042][1][info][main] [source/server/server.cc:950] all clusters initialized. initializing init manager
[2024-04-26 02:51:18.071][1][error][misc] [source/extensions/common/aws/utility.cc:501] Unable to retrieve integer value from json: Expiration
[2024-04-26 02:51:18.071][1][warning][aws] [source/extensions/common/aws/credentials_provider_impl.cc:700] Could not get Expiration value of AWS credentials document from STS, so setting expiration to 1 hour in future
[2024-04-26 02:51:18.133][1][error][misc] [source/extensions/common/aws/utility.cc:501] Unable to retrieve integer value from json: Expiration
[2024-04-26 02:51:18.133][1][warning][aws] [source/extensions/common/aws/credentials_provider_impl.cc:700] Could not get Expiration value of AWS credentials document from STS, so setting expiration to 1 hour in future
[2024-04-26 02:51:18.133][1][info][config] [source/common/listener_manager/listener_manager_impl.cc:930] all dependencies initialized. starting workers
[2024-04-26 03:06:18.132][1][info][main] [source/server/drain_manager_impl.cc:208] shutting down parent after drain```
@MalibuKoKo We retrieve this from STS using json format which does in fact use the unix timestamp. However STS is returning a scientific notation version in the json payload. I'm investigating with the STS team as to whether this is accurate and then can propose a fix.
It's not expected behaviour from STS, but valid json nontheless. I have a fix for this that will go to PR shortly.
PR https://github.com/envoyproxy/envoy/pull/34426 fixes this issue. Apologies for the delay, the fix was dependent on https://github.com/envoyproxy/envoy/pull/34138 merged today
@nbaws: thank you very much for the correction I will test the fix in the next version.