envoy icon indicating copy to clipboard operation
envoy copied to clipboard

tls/validation_context_sds_secret_config does not work with ADS

Open belyalov opened this issue 3 years ago • 8 comments

Title: tls/validation_context_sds_secret_config does not work when configured with ADS

Description: When configuring UpstreamTlsContext/common_tls_context with validation_context_sds_secret_config set to fetch secrets from ADS envoy fails to start (crashes)

Repro steps: Use config below (literally you can use any config but use validation_context_sds_secret_config on upstream cluster:

static_resources: {
  listeners: {
    name: "http_forward_proxy"
    address: {
      socket_address: {
        address: "127.0.0.1"
        port_value: 80
      }
    }
    filter_chains: {
      filters: {
        name: "envoy.filters.network.http_connection_manager"
        typed_config: {
          [type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager]: {
            stat_prefix: "http_forward_proxy"
            route_config: {
              name: "catch_all_static_routes"
              virtual_hosts: {
                name: "catch_all"
                domains: "*"
                routes: {
                  match: {
                    prefix: "/"
                  }
                  route: {
                    cluster: "localhost"
                  }
                }
              }
            }
            http_filters: {
              name: "envoy.filters.http.router"
            }
          }
        }
      }
    }
  }
  clusters: {
    name: "localhost"
    type: STATIC
    load_assignment: {
      cluster_name: "localhost"
      endpoints: {
        lb_endpoints: {
          endpoint: {
            address: {
              socket_address: {
                address: "127.0.0.1"
                port_value: 8080
              }
            }
          }
        }
      }
    }
    transport_socket: {
      name: "envoy.transport_sockets.tls"
      typed_config: {
        [type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext]: {
          common_tls_context: {
            validation_context_sds_secret_config: {
              name: "something"
              sds_config: {
                ads: {}
                resource_api_version: V3
              }
            }
          }
        }
      }
    }
  }
  clusters: {
    name: "envoy_xds"
    load_assignment: {
      cluster_name: "envoy_xds"
      endpoints: {
        lb_endpoints: {
          endpoint: {
            address: {
              pipe: {
                path: "/tmp/envoy_xds.sock"
              }
            }
          }
        }
      }
    }
    typed_extension_protocol_options: {
      key: "envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
      value: {
        [type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions]: {
          explicit_http_config: {
            http2_protocol_options: {}
          }
        }
      }
    }
  }
}
dynamic_resources: {
  ads_config: {
    api_type: GRPC
    transport_api_version: V3
    grpc_services: {
      envoy_grpc: {
        cluster_name: "envoy_xds"
      }
    }
    set_node_on_first_message_only: true
  }
}

Logs and Call Stack:

[2022-01-19 14:43:22.326][5264002][info][main] [external/envoy/source/server/server.cc:381] initializing epoch 0 (base id=0, hot restart version=disabled)
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:383] statically linked extensions:
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.http.stateful_session: envoy.http.stateful_session.cookie
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.dubbo_proxy.protocols: dubbo
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.dubbo_proxy.route_matchers: default
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.formatter: envoy.formatter.metadata, envoy.formatter.req_without_query
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.common.key_value: envoy.key_value.file_based
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.cors, envoy.filters.http.csrf, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.dynamo, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.http_dynamo_filter, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router, match-wrapper
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.tracers: envoy.dynamic.ot, envoy.lightstep, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.lightstep, envoy.tracers.opencensus, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.filters.network: envoy.client_ssl_auth, envoy.echo, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.http.stateful_header_formatters: preserve_case
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.upstreams.http.http_protocol_options
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.matching.http.input: request-headers, request-trailers, response-headers, response-trailers
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.http.cache: envoy.extensions.http.cache.simple
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.access_logger.extension_filters: envoy.access_loggers.extension_filters.cel
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.bootstrap: envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.network.dns_resolver: envoy.network.dns_resolver.apple, envoy.network.dns_resolver.cares
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.request_id: envoy.request_id.uuid
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.resolvers: envoy.ip
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.matching.input_matchers: envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.health_checkers: envoy.health_checkers.redis
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2022-01-19 14:43:22.327][5264002][info][main] [external/envoy/source/server/server.cc:385]   envoy.matching.action: composite-action, skip
[2022-01-19 14:43:22.336][5264002][info][main] [external/envoy/source/server/server.cc:433] HTTP header map info:
[2022-01-19 14:43:22.338][5264002][debug][runtime] [external/envoy/source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2022-01-19 14:43:22.338][5264002][debug][runtime] [external/envoy/source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2022-01-19 14:43:22.339][5264002][debug][runtime] [external/envoy/source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2022-01-19 14:43:22.339][5264002][debug][runtime] [external/envoy/source/common/runtime/runtime_features.cc:31] Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
[2022-01-19 14:43:22.339][5264002][info][main] [external/envoy/source/server/server.cc:436]   request header map: 640 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-method,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-proto,x-ot-span-context,x-request-id
[2022-01-19 14:43:22.339][5264002][info][main] [external/envoy/source/server/server.cc:436]   request trailer map: 128 bytes: 
[2022-01-19 14:43:22.339][5264002][info][main] [external/envoy/source/server/server.cc:436]   response header map: 424 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2022-01-19 14:43:22.339][5264002][info][main] [external/envoy/source/server/server.cc:436]   response trailer map: 152 bytes: grpc-message,grpc-status
[2022-01-19 14:43:22.345][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.shrink_heap.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reduce_timeouts.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reset_high_memory_stream.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reduce_timeouts.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reset_high_memory_stream.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reduce_timeouts.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reset_high_memory_stream.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reduce_timeouts.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reset_high_memory_stream.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reduce_timeouts.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reset_high_memory_stream.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reduce_timeouts.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reset_high_memory_stream.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reduce_timeouts.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reset_high_memory_stream.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reduce_timeouts.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reset_high_memory_stream.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reduce_timeouts.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reset_high_memory_stream.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reduce_timeouts.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reject_incoming_connections.
[2022-01-19 14:43:22.346][5264002][debug][main] [external/envoy/source/server/overload_manager_impl.cc:448] No overload action is configured for envoy.overload_actions.reset_high_memory_stream.
[2022-01-19 14:43:22.347][5264002][info][main] [external/envoy/source/server/server.cc:772] runtime: {}
[2022-01-19 14:43:22.347][5264002][warning][main] [external/envoy/source/server/server.cc:629] No admin address given, so no admin HTTP server started.
[2022-01-19 14:43:22.348][5264002][debug][misc] [external/envoy/source/common/network/dns_resolver/dns_factory_util.cc:42] create Apple DNS resolver type: envoy.network.dns_resolver.apple in MacOS.
[2022-01-19 14:43:22.348][5264002][debug][misc] [external/envoy/source/common/network/dns_resolver/dns_factory_util.cc:81] create DNS resolver type: envoy.network.dns_resolver.apple
[2022-01-19 14:43:22.348][5264002][info][config] [external/envoy/source/server/configuration_impl.cc:127] loading tracing configuration
[2022-01-19 14:43:22.348][5264002][info][config] [external/envoy/source/server/configuration_impl.cc:87] loading 0 static secret(s)
[2022-01-19 14:43:22.348][5264002][info][config] [external/envoy/source/server/configuration_impl.cc:93] loading 2 cluster(s)
[2022-01-19 14:43:22.349][5264015][debug][grpc] [external/envoy/source/common/grpc/google_async_client_impl.cc:51] completionThread running
[2022-01-19 14:43:22.350][5264002][debug][init] [external/envoy/source/common/init/manager_impl.cc:24] added shared target SdsApi something to init manager Cluster localhost
[2022-01-19 14:43:22.351][5264002][debug][config] [external/envoy/source/extensions/transport_sockets/tls/context_config_impl.cc:255] API3 0x60000296efd0
[2022-01-19 14:43:22.355][5264002][debug][upstream] [external/envoy/source/common/upstream/upstream_impl.cc:256] transport socket match, socket default selected for host with address 127.0.0.1:8080
[2022-01-19 14:43:22.361][5264002][debug][upstream] [external/envoy/source/common/upstream/upstream_impl.cc:256] transport socket match, socket default selected for host with address /tmp/envoy_xds.sock
[2022-01-19 14:43:22.361][5264002][debug][upstream] [external/envoy/source/common/upstream/upstream_impl.cc:1183] initializing Primary cluster envoy_xds completed
[2022-01-19 14:43:22.361][5264002][debug][init] [external/envoy/source/common/init/manager_impl.cc:49] init manager Cluster envoy_xds contains no targets
[2022-01-19 14:43:22.361][5264002][debug][init] [external/envoy/source/common/init/watcher_impl.cc:14] init manager Cluster envoy_xds initialized, notifying ClusterImplBase
[2022-01-19 14:43:22.361][5264002][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1053] adding TLS cluster envoy_xds
[2022-01-19 14:43:22.362][5264002][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:1116] membership update for TLS cluster envoy_xds added 1 removed 0
[2022-01-19 14:43:22.362][5264002][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:135] cm init: init complete: cluster=envoy_xds primary=0 secondary=0
[2022-01-19 14:43:22.362][5264002][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:155] maybe finish initialize state: 0
[2022-01-19 14:43:22.362][5264002][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:101] cm init: adding: cluster=envoy_xds primary=0 secondary=0
[2022-01-19 14:43:22.362][5264002][debug][upstream] [external/envoy/source/common/upstream/upstream_impl.cc:1183] initializing Primary cluster localhost completed
[2022-01-19 14:43:22.362][5264002][debug][init] [external/envoy/source/common/init/manager_impl.cc:53] init manager Cluster localhost initializing
[2022-01-19 14:43:22.362][5264002][debug][init] [external/envoy/source/common/init/target_impl.cc:15] init manager Cluster localhost initializing shared target SdsApi something
[2022-01-19 14:43:22.362][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:104] Caught Segmentation fault: 11, suspect faulting address 0x0
[2022-01-19 14:43:22.362][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-01-19 14:43:22.362][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:92] Envoy version: 53c94ce2f3aa3ede2868372a56373504df31ad3f/1.21.0/Modified/DEBUG/BoringSSL
[2022-01-19 14:43:22.366][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x109dda79c]
[2022-01-19 14:43:22.366][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #1: _sigtramp [0x191ac04e4]
[2022-01-19 14:43:22.369][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #2: Envoy::Config::GrpcSubscriptionImpl::start() [0x108d80b44]
[2022-01-19 14:43:22.370][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #3: Envoy::Secret::SdsApi::initialize() [0x10987c574]
[2022-01-19 14:43:22.371][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #4: Envoy::Secret::SdsApi::SdsApi()::$_0::operator()() [0x10988d050]
[2022-01-19 14:43:22.373][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #5: std::__1::__invoke<>() [0x10988d014]
[2022-01-19 14:43:22.374][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #6: std::__1::__invoke_void_return_wrapper<>::__call<>() [0x10988cfc8]
[2022-01-19 14:43:22.375][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #7: std::__1::__function::__alloc_func<>::operator()() [0x10988cfa0]
[2022-01-19 14:43:22.377][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #8: std::__1::__function::__func<>::operator()() [0x10988bb4c]
[2022-01-19 14:43:22.378][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #9: std::__1::__function::__value_func<>::operator()() [0x1049b6fdc]
[2022-01-19 14:43:22.379][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #10: std::__1::function<>::operator()() [0x1049b6d08]
[2022-01-19 14:43:22.380][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #11: std::__1::__invoke<>() [0x109c14768]
[2022-01-19 14:43:22.381][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #12: std::__1::__call_once_param<>::__execute<>() [0x109c14740]
[2022-01-19 14:43:22.383][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #13: std::__1::__call_once_param<>::operator()() [0x109c14710]
[2022-01-19 14:43:22.384][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #14: std::__1::__call_once_proxy<>() [0x109c145e8]
[2022-01-19 14:43:22.384][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #15: std::__1::__call_once() [0x1919fa440]
[2022-01-19 14:43:22.385][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #16: std::__1::call_once<>() [0x109c134d4]
[2022-01-19 14:43:22.387][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #17: Envoy::Init::SharedTargetImpl::SharedTargetImpl()::$_1::operator()() [0x109c133f8]
[2022-01-19 14:43:22.388][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #18: std::__1::__invoke<>() [0x109c1333c]
[2022-01-19 14:43:22.389][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #19: std::__1::__invoke_void_return_wrapper<>::__call<>() [0x109c132c8]
[2022-01-19 14:43:22.391][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #20: std::__1::__function::__alloc_func<>::operator()() [0x109c13288]
[2022-01-19 14:43:22.392][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #21: std::__1::__function::__func<>::operator()() [0x109c11e98]
[2022-01-19 14:43:22.393][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #22: std::__1::__function::__value_func<>::operator()() [0x109c0da98]
[2022-01-19 14:43:22.395][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #23: std::__1::function<>::operator()() [0x109c09b0c]
[2022-01-19 14:43:22.396][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #24: Envoy::Init::TargetHandleImpl::initialize() [0x109c09764]
[2022-01-19 14:43:22.397][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #25: Envoy::Init::ManagerImpl::initialize() [0x109c007f8]
[2022-01-19 14:43:22.399][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #26: Envoy::Upstream::ClusterImplBase::onPreInitComplete() [0x108325704]
[2022-01-19 14:43:22.400][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #27: Envoy::Upstream::StaticClusterImpl::startPreInit() [0x108482474]
[2022-01-19 14:43:22.401][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #28: Envoy::Upstream::ClusterImplBase::initialize() [0x108325408]
[2022-01-19 14:43:22.403][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #29: Envoy::Upstream::ClusterManagerInitHelper::addCluster() [0x107b68790]
[2022-01-19 14:43:22.404][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #30: Envoy::Upstream::ClusterManagerImpl::ClusterManagerImpl() [0x107b6d600]
[2022-01-19 14:43:22.405][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #31: Envoy::Upstream::ClusterManagerImpl::ClusterManagerImpl() [0x107b71080]
[2022-01-19 14:43:22.406][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #32: Envoy::Upstream::ProdClusterManagerFactory::clusterManagerFromProto() [0x107b7f064]
[2022-01-19 14:43:22.408][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #33: Envoy::Server::Configuration::MainImpl::initialize() [0x108970248]
[2022-01-19 14:43:22.409][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #34: Envoy::Server::InstanceImpl::initialize() [0x1079bf3d8]
[2022-01-19 14:43:22.410][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #35: Envoy::Server::InstanceImpl::InstanceImpl() [0x1079bad70]
[2022-01-19 14:43:22.411][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #36: Envoy::Server::InstanceImpl::InstanceImpl() [0x1079c02e4]
[2022-01-19 14:43:22.413][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #37: std::__1::make_unique<>() [0x104699e58]
[2022-01-19 14:43:22.414][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #38: Envoy::MainCommonBase::MainCommonBase() [0x1046990c8]
[2022-01-19 14:43:22.415][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #39: Envoy::MainCommonBase::MainCommonBase() [0x10469a2a4]
[2022-01-19 14:43:22.416][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #40: Envoy::MainCommon::MainCommon() [0x10469b1d8]
[2022-01-19 14:43:22.417][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #41: Envoy::MainCommon::MainCommon() [0x10469b338]
[2022-01-19 14:43:22.418][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #42: std::__1::make_unique<>() [0x10469b908]
[2022-01-19 14:43:22.419][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #43: Envoy::MainCommon::main() [0x10469b6fc]
[2022-01-19 14:43:22.420][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #44: main [0x1046941d8]
[2022-01-19 14:43:22.420][5264002][critical][backtrace] [external/envoy/source/server/backtrace.h:96] #45: start [0x1195890f4]

belyalov avatar Jan 19 '22 21:01 belyalov

Thanks for reporting. This sounds like a bug from the initialization order because when the static cluster initializes the ADS subscription doesn't exist yet.

lizan avatar Jan 20 '22 21:01 lizan

Check https://github.com/envoyproxy/envoy/blob/main/test/integration/ads_integration_test.cc#L177-L188, tls/validation_context_sds_secret_config can work with ADS.

daixiang0 avatar Jan 25 '22 04:01 daixiang0

I see the test, @daixiang0 But as you can see from the output - it real run it crashes. Did I misconfigured something?

belyalov avatar Jan 25 '22 17:01 belyalov

@lizan is it something that relatively easy to fix? I so - I can help here if you point me to some places :)

belyalov avatar Jun 06 '22 22:06 belyalov

@belyalov have you been able to resolve this issue?

ddymko avatar Aug 10 '22 17:08 ddymko

@ddymko unfortunately no..(

belyalov avatar Aug 10 '22 17:08 belyalov

cc @adisuissa for attention.

kyessenov avatar Aug 10 '22 20:08 kyessenov

From what I gather it's probably because the static cluster "localhost" is configured to use SDS (on top of ADS), but ADS is probably initialized after the static clusters are initialized. If "localhost" was a dynamically added cluster (CDS), then it will probably work. One way to test this is to configure a file-based CDS input, and add the "localhost" cluster using that.

adisuissa avatar Aug 10 '22 21:08 adisuissa