"Remote Reset 2" when using envoy with Python gRPC Services via UDS Channels

Open thinkingotter opened this issue 1 year ago • 3 comments

Title: One line description We are experiencing intermittent remote resets with Envoy and Python gRPC services over UDS channels

Description: We're encountering intermittent remote resets errors when using Envoy with Python gRPC services connected via UDS channels. Specifically, a client sending health check requests every 30 seconds consistently encounters this issue when creating a new channel for each request. client side error observed: 728164723 {0.434192419052124} <Result port_grpc_21519 result=False service=<Service code-health-policy-engine-grpc+i001 1d6d126bf13db575dacd354b003cc6da5562e2f0> extra={"secure": "UNAVAILABLE: upstream connect error or disconnect/reset before headers. reset reason: remote reset", "insecure": "UNAVAILABLE: failed to connect to all addresses; last error: UNAVAILABLE: ipv4:10.205.1.173:21519: Socket closed", "oor": false}>

Key Observations:

No errors with Rust services: We do not observe this issue with Rust gRPC services deployed with Envoy via UDS.
No errors if the client reuse the channel cross subsequent requests.
Error frequency: The error occurs approximately once per hour.
Envoy trace logs: Envoy logs indicate a "remote reset" from the upstream server. Error code: The error code associated with the remote reset is 2, suggesting a protocol error or unexpected behavior on the upstream service. Repro steps:

Include sample requests, environment, etc. All data and inputs required to reproduce the bug. python gRPC service connected with envoy via UDS:

client request:

 while True:
    with grpc.secure_channel(url, cred) as channel:
        stub = HealthStub(channel)
        req = HealthCheckRequest(service='')
        try:
            res = stub.Check(req, timeout=timeout)
            success += 1
            counter += 1
            print(f'{counter=}, {success=}, {failure=}')
        except grpc.RpcError as ex:
            code, details = ex.code(), ex.details()
            failure += 1
            counter += 1
            print(f'{counter=}, {success=}, {failure=}, {code=}, {details=}')

class HealthStub(object):
    """Missing associated documentation comment in .proto file."""

    def __init__(self, channel):
        """Constructor.

        Args:
            channel: A grpc.Channel.
        """
        self.Check = channel.unary_unary(
                '/grpc.health.v1.Health/Check',
                request_serializer=obhc_dot_messages_dot_grpc__health__v1__pb2.HealthCheckRequest.SerializeToString,
                response_deserializer=obhc_dot_messages_dot_grpc__health__v1__pb2.HealthCheckResponse.FromString,
                )

Note: The Envoy_collect tool gathers a tarball with debug logs, config and the following admin endpoints: /stats, /clusters and /server_info. Please note if there are privacy concerns, sanitize the data prior to sharing the tarball/pasting.

Admin and Stats Output:

Include the admin output for the following endpoints: /stats, /clusters, /routes, /server_info. For more information, refer to the admin endpoint documentation.

/stats: {"stats":[{"name":"cluster_manager.cds.version_text","value":""},{"name":"cluster_manager.odcds.version_text","value":""},{"name":"control_plane.identifier","value":""},{"name":"http .outbound_http.rds.global.version_text","value":"0"},{"name":"http.outbound_http.rds.vhds.global.version_text","value":""},{"name":"listener_manager.lds.version_text","value":""},{"n ame":"sds.tls_sds.version_text","value":""},{"name":"sds.validation_context_sds.version_text","value":""},{"name":"cluster.uds_inbound_local_app_cluster.assignment_stale","value":0}, {"name":"cluster.uds_inbound_local_app_cluster.assignment_timeout_received","value":0},{"name":"cluster.uds_inbound_local_app_cluster.bind_errors","value":0},{"name":"cluster.uds_inb ound_local_app_cluster.circuit_breakers.default.cx_open","value":0},{"name":"cluster.uds_inbound_local_app_cluster.circuit_breakers.default.cx_pool_open","value":0},{"name":"cluster. uds_inbound_local_app_cluster.circuit_breakers.default.rq_open","value":0},{"name":"cluster.uds_inbound_local_app_cluster.circuit_breakers.default.rq_pending_open","value":0},{"name" :"cluster.uds_inbound_local_app_cluster.circuit_breakers.default.rq_retry_open","value":0},{"name":"cluster.uds_inbound_local_app_cluster.circuit_breakers.high.cx_open","value":0},{" name":"cluster.uds_inbound_local_app_cluster.circuit_breakers.high.cx_pool_open","value":0},{"name":"cluster.uds_inbound_local_app_cluster.circuit_breakers.high.rq_open","value":0},{ "name":"cluster.uds_inbound_local_app_cluster.circuit_breakers.high.rq_pending_open","value":0},{"name":"cluster.uds_inbound_local_app_cluster.circuit_breakers.high.rq_retry_open","v alue":0},{"name":"cluster.uds_inbound_local_app_cluster.default.total_match_count","value":1},{"name":"cluster.uds_inbound_local_app_cluster.lb_healthy_panic","value":0},{"name":"clu ster.uds_inbound_local_app_cluster.lb_local_cluster_not_ok","value":0},{"name":"cluster.uds_inbound_local_app_cluster.lb_recalculate_zone_structures","value":0},{"name":"cluster.uds_ inbound_local_app_cluster.lb_subsets_active","value":0},{"name":"cluster.uds_inbound_local_app_cluster.lb_subsets_created","value":0},{"name":"cluster.uds_inbound_local_app_cluster.l b_subsets_fallback","value":0},{"name":"cluster.uds_inbound_local_app_cluster.lb_subsets_fallback_panic","value":0},{"name":"cluster.uds_inbound_local_app_cluster.lb_subsets_removed" ,"value":0},{"name":"cluster.uds_inbound_local_app_cluster.lb_subsets_selected","value":0},{"name":"cluster.uds_inbound_local_app_cluster.lb_zone_cluster_too_small","value":0},{"name ":"cluster.uds_inbound_local_app_cluster.lb_zone_no_capacity_left","value":0},{"name":"cluster.uds_inbound_local_app_cluster.lb_zone_number_differs","value":0},{"name":"cluster.uds_i nbound_local_app_cluster.lb_zone_routing_all_directly","value":0},{"name":"cluster.uds_inbound_local_app_cluster.lb_zone_routing_cross_zone","value":0},{"name":"cluster.uds_inbound_l ocal_app_cluster.lb_zone_routing_sampled","value":0},{"name":"cluster.uds_inbound_local_app_cluster.max_host_weight","value":0},{"name":"cluster.uds_inbound_local_app_cluster.members hip_change","value":1},{"name":"cluster.uds_inbound_local_app_cluster.membership_degraded","value":0},{"name":"cluster.uds_inbound_local_app_cluster.membership_excluded","value":0},{ "name":"cluster.uds_inbound_local_app_cluster.membership_healthy","value":1},{"name":"cluster.uds_inbound_local_app_cluster.membership_total","value":1},{"name":"cluster.uds_inbound_ local_app_cluster.original_dst_host_invalid","value":0},{"name":"cluster.uds_inbound_local_app_cluster.retry_or_shadow_abandoned","value":0},{"name":"cluster.uds_inbound_local_app_cl uster.update_attempt","value":0},{"name":"cluster.uds_inbound_local_app_cluster.update_empty","value":0},{"name":"cluster.uds_inbound_local_app_cluster.update_failure","value":0},{"n ame":"cluster.uds_inbound_local_app_cluster.update_no_rebuild","value":0},{"name":"cluster.uds_inbound_local_app_cluster.update_success","value":0},{"name":"cluster.uds_inbound_local app_cluster.upstream_cx_active","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_close_notify","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstrea m_cx_connect_attempts_exceeded","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_connect_fail","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream cx_connect_timeout","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_connect_with_0_rtt","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_d estroy","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_destroy_local","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_destroy_local_with active_rq","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_destroy_remote","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_destroy_remote with_active_rq","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_destroy_with_active_rq","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_ht tp1_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_http2_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_http3_total","valu e":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_idle_timeout","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_max_duration_reached","value":0}, {"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_max_requests","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_none_healthy","value":0},{"name":"clust er.uds_inbound_local_app_cluster.upstream_cx_overflow","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_pool_overflow","value":0},{"name":"cluster.uds_inbound_lo cal_app_cluster.upstream_cx_protocol_error","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_rx_bytes_buffered","value":0},{"name":"cluster.uds_inbound_local_app cluster.upstream_cx_rx_bytes_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_c x_tx_bytes_buffered","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_tx_bytes_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_flow_con trol_backed_up_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_flow_control_drained_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstrea m_flow_control_paused_reading_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_flow_control_resumed_reading_total","value":0},{"name":"cluster.uds_inbound_lo cal_app_cluster.upstream_http3_broken","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_internal_redirect_failed_total","value":0},{"name":"cluster.uds_inbound_loca l_app_cluster.upstream_internal_redirect_succeeded_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_0rtt","value":0},{"name":"cluster.uds_inbound_local_ap p_cluster.upstream_rq_active","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_cancelled","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_c ompleted","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_maintenance_mode","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_max_duration_r eached","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_pending_active","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_pending_failure_ej ect","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_pending_overflow","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_pending_total","val ue":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_per_try_idle_timeout","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_per_try_timeout","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_retry","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_retry_backoff_exponential","value":0},{"na me":"cluster.uds_inbound_local_app_cluster.upstream_rq_retry_backoff_ratelimited","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_retry_limit_exceeded","value": 0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_retry_overflow","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_retry_success","value":0},{"name": "cluster.uds_inbound_local_app_cluster.upstream_rq_rx_reset","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_timeout","value":0},{"name":"cluster.uds_inbound_lo cal_app_cluster.upstream_rq_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster.upstream_rq_tx_reset","value":0},{"name":"cluster.uds_inbound_local_app_cluster.version", "value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.assignment_stale","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.assignment_timeout_received","value":0 },{"name":"cluster.uds_inbound_local_app_cluster_grpc.bind_errors","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.circuit_breakers.default.cx_open","value":0},{"name" :"cluster.uds_inbound_local_app_cluster_grpc.circuit_breakers.default.cx_pool_open","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.circuit_breakers.default.rq_open"," value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.circuit_breakers.default.rq_pending_open","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.circuit_breaker s.default.rq_retry_open","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.circuit_breakers.high.cx_open","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc. circuit_breakers.high.cx_pool_open","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.circuit_breakers.high.rq_open","value":0},{"name":"cluster.uds_inbound_local_app_cl uster_grpc.circuit_breakers.high.rq_pending_open","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.circuit_breakers.high.rq_retry_open","value":0},{"name":"cluster.uds inbound_local_app_cluster_grpc.default.total_match_count","value":1},{"name":"cluster.uds_inbound_local_app_cluster_grpc.external.upstream_rq_200","value":3315935},{"name":"cluster.u ds_inbound_local_app_cluster_grpc.external.upstream_rq_2xx","value":3315935},{"name":"cluster.uds_inbound_local_app_cluster_grpc.external.upstream_rq_503","value":74299},{"name":"clu ster.uds_inbound_local_app_cluster_grpc.external.upstream_rq_5xx","value":74299},{"name":"cluster.uds_inbound_local_app_cluster_grpc.external.upstream_rq_completed","value":3390234}, {"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.deferred_stream_close","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.dropped_headers_with_underscores ","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.header_overflow","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.headers_cb_no_stream","va lue":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.inbound_empty_frames_flood","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.inbound_priority_fr ames_flood","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.inbound_window_update_frames_flood","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.ht tp2.keepalive_timeout","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.metadata_empty_frames","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http 2.outbound_control_flood","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.outbound_flood","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.pe nding_send_bytes","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.requests_rejected_with_underscores_in_headers","value":0},{"name":"cluster.uds_inbound_local_ap p_cluster_grpc.http2.rx_messaging_error","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.rx_reset","value":74299},{"name":"cluster.uds_inbound_local_app_cluster_ grpc.http2.stream_refused_errors","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.streams_active","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc. http2.trailers","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.tx_flush_timeout","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.http2.tx_reset", "value":1649},{"name":"cluster.uds_inbound_local_app_cluster_grpc.lb_healthy_panic","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.lb_local_cluster_not_ok","value":0} ,{"name":"cluster.uds_inbound_local_app_cluster_grpc.lb_recalculate_zone_structures","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.lb_subsets_active","value":0},{"na me":"cluster.uds_inbound_local_app_cluster_grpc.lb_subsets_created","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.lb_subsets_fallback","value":0},{"name":"cluster.ud s_inbound_local_app_cluster_grpc.lb_subsets_fallback_panic","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.lb_subsets_removed","value":0},{"name":"cluster.uds_inbound local_app_cluster_grpc.lb_subsets_selected","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.lb_zone_cluster_too_small","value":0},{"name":"cluster.uds_inbound_local_a pp_cluster_grpc.lb_zone_no_capacity_left","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.lb_zone_number_differs","value":0},{"name":"cluster.uds_inbound_local_app_clu ster_grpc.lb_zone_routing_all_directly","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.lb_zone_routing_cross_zone","value":0},{"name":"cluster.uds_inbound_local_app_c luster_grpc.lb_zone_routing_sampled","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.max_host_weight","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.me mbership_change","value":1},{"name":"cluster.uds_inbound_local_app_cluster_grpc.membership_degraded","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.membership_exclude d","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.membership_healthy","value":1},{"name":"cluster.uds_inbound_local_app_cluster_grpc.membership_total","value":1},{"na me":"cluster.uds_inbound_local_app_cluster_grpc.original_dst_host_invalid","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.retry_or_shadow_abandoned","value":0},{"name ":"cluster.uds_inbound_local_app_cluster_grpc.update_attempt","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.update_empty","value":0},{"name":"cluster.uds_inbound_loc al_app_cluster_grpc.update_failure","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.update_no_rebuild","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.u pdate_success","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_active","value":40},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_close_no tify","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_connect_attempts_exceeded","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx connect_fail","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_connect_timeout","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx connect_with_0_rtt","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_destroy","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_des troy_local","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_destroy_local_with_active_rq","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.up stream_cx_destroy_remote","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_destroy_remote_with_active_rq","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_destroy_with_active_rq","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_http1_total","value":0},{"name":"cluster.uds_inbound_local app_cluster_grpc.upstream_cx_http2_total","value":40},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_http3_total","value":0},{"name":"cluster.uds_inbound_local_app cluster_grpc.upstream_cx_idle_timeout","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_max_duration_reached","value":0},{"name":"cluster.uds_inbound_local app_cluster_grpc.upstream_cx_max_requests","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_none_healthy","value":0},{"name":"cluster.uds_inbound_local_app_ cluster_grpc.upstream_cx_overflow","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_pool_overflow","value":0},{"name":"cluster.uds_inbound_local_app_cluster grpc.upstream_cx_protocol_error","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_rx_bytes_buffered","value":2052},{"name":"cluster.uds_inbound_local_app_c luster_grpc.upstream_cx_rx_bytes_total","value":246271814},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_total","value":40},{"name":"cluster.uds_inbound_local_app_c luster_grpc.upstream_cx_tx_bytes_buffered","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_tx_bytes_total","value":2952467002},{"name":"cluster.uds_inbound local_app_cluster_grpc.upstream_flow_control_backed_up_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_flow_control_drained_total","value":0},{"name": "cluster.uds_inbound_local_app_cluster_grpc.upstream_flow_control_paused_reading_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_flow_control_resumed_r eading_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_http3_broken","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_internal_r edirect_failed_total","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_internal_redirect_succeeded_total","value":0},{"name":"cluster.uds_inbound_local_app_clu ster_grpc.upstream_rq_0rtt","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_200","value":3315935},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstr eam_rq_2xx","value":3315935},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_503","value":74299},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_5xx", "value":74299},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_active","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_cancelled","value":0 },{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_completed","value":3390234},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_maintenance_mode","value ":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_max_duration_reached","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_pending_active", "value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_pending_failure_eject","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_pending_o verflow","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_pending_total","value":40},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_per_try idle_timeout","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_per_try_timeout","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq retry","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_retry_backoff_exponential","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_r q_retry_backoff_ratelimited","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_retry_limit_exceeded","value":0},{"name":"cluster.uds_inbound_local_app_cluste r_grpc.upstream_rq_retry_overflow","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_retry_success","value":0},{"name":"cluster.uds_inbound_local_app_cluster grpc.upstream_rq_rx_reset","value":74299},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_timeout","value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.ups tream_rq_total","value":3391883},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_tx_reset","value":1649},{"name":"cluster.uds_inbound_local_app_cluster_grpc.version", "value":0},{"name":"cluster.uds_inbound_local_app_cluster_grpc.zone.corp-lca1..upstream_rq_200","value":3315935},{"name":"cluster.uds_inbound_local_app_cluster_grpc.zone.corp-lca1..u pstream_rq_2xx","value":3315935},{"name":"cluster.uds_inbound_local_app_cluster_grpc.zone.corp-lca1..upstream_rq_503","value":74299},{"name":"cluster.uds_inbound_local_app_cluster_gr pc.zone.corp-lca1..upstream_rq_5xx","value":74299},{"name":"cluster.uds_inbound_local_app_cluster_grpc.zone.corp-lca1..upstream_rq_completed","value":3390234},{"name":"cluster.xds_cl uster.assignment_stale","value":0},{"name":"cluster.xds_cluster.assignment_timeout_received","value":0},{"name":"cluster.xds_cluster.bind_errors","value":0},{"name":"cluster.xds_clus ter.circuit_breakers.default.cx_open","value":0},{"name":"cluster.xds_cluster.circuit_breakers.default.cx_pool_open","value":0},{"name":"cluster.xds_cluster.circuit_breakers.default. rq_open","value":0},{"name":"cluster.xds_cluster.circuit_breakers.default.rq_pending_open","value":0},{"name":"cluster.xds_cluster.circuit_breakers.default.rq_retry_open","value":0}, {"name":"cluster.xds_cluster.circuit_breakers.high.cx_open","value":0},{"name":"cluster.xds_cluster.circuit_breakers.high.cx_pool_open","value":0},{"name":"cluster.xds_cluster.circui t_breakers.high.rq_open","value":0},{"name":"cluster.xds_cluster.circuit_breakers.high.rq_pending_open","value":0},{"name":"cluster.xds_cluster.circuit_breakers.high.rq_retry_open"," value":0},{"name":"cluster.xds_cluster.client_ssl_socket_factory.downstream_context_secrets_not_ready","value":0},{"name":"cluster.xds_cluster.client_ssl_socket_factory.ssl_context_u pdate_by_sds","value":2},{"name":"cluster.xds_cluster.client_ssl_socket_factory.upstream_context_secrets_not_ready","value":0},{"name":"cluster.xds_cluster.default.total_match_count" ,"value":2079796},{"name":"cluster.xds_cluster.http2.deferred_stream_close","value":0},{"name":"cluster.xds_cluster.http2.dropped_headers_with_underscores","value":0},{"name":"cluste r.xds_cluster.http2.header_overflow","value":0},{"name":"cluster.xds_cluster.http2.headers_cb_no_stream","value":0},{"name":"cluster.xds_cluster.http2.inbound_empty_frames_flood","va lue":0},{"name":"cluster.xds_cluster.http2.inbound_priority_frames_flood","value":0},{"name":"cluster.xds_cluster.http2.inbound_window_update_frames_flood","value":0},{"name":"cluste r.xds_cluster.http2.keepalive_timeout","value":0},{"name":"cluster.xds_cluster.http2.metadata_empty_frames","value":0},{"name":"cluster.xds_cluster.http2.outbound_control_flood","val ue":0},{"name":"cluster.xds_cluster.http2.outbound_flood","value":0},{"name":"cluster.xds_cluster.http2.pending_send_bytes","value":0},{"name":"cluster.xds_cluster.http2.requests_rej ected_with_underscores_in_headers","value":0},{"name":"cluster.xds_cluster.http2.rx_messaging_error","value":0},{"name":"cluster.xds_cluster.http2.rx_reset","value":0},{"name":"clust er.xds_cluster.http2.stream_refused_errors","value":0},{"name":"cluster.xds_cluster.http2.streams_active","value":1},{"name":"cluster.xds_cluster.http2.trailers","value":0},{"name":" cluster.xds_cluster.http2.tx_flush_timeout","value":0},{"name":"cluster.xds_cluster.http2.tx_reset","value":0},{"name":"cluster.xds_cluster.lb_healthy_panic","value":0},{"name":"clus ter.xds_cluster.lb_local_cluster_not_ok","value":0},{"name":"cluster.xds_cluster.lb_recalculate_zone_structures","value":0},{"name":"cluster.xds_cluster.lb_subsets_active","value":0},{"name":"cluster.xds_cluster.lb_subsets_created","value":0},{"name":"cluster.xds_cluster.lb_subsets_fallback","value":0},{"name":"cluster.xds_cluster.lb_subsets_fallback_panic","val ue":0},{"name":"cluster.xds_cluster.lb_subsets_removed","value":0},{"name":"cluster.xds_cluster.lb_subsets_selected","value":0},{"name":"cluster.xds_cluster.lb_zone_cluster_too_small ","value":0},{"name":"cluster.xds_cluster.lb_zone_no_capacity_left","value":0},{"name":"cluster.xds_cluster.lb_zone_number_differs","value":0},{"name":"cluster.xds_cluster.lb_zone_ro uting_all_directly","value":0},{"name":"cluster.xds_cluster.lb_zone_routing_cross_zone","value":0},{"name":"cluster.xds_cluster.lb_zone_routing_sampled","value":0},{"name":"cluster.x ds_cluster.max_host_weight","value":1},{"name":"cluster.xds_cluster.membership_change","value":213},{"name":"cluster.xds_cluster.membership_degraded","value":0},{"name":"cluster.xds cluster.membership_excluded","value":0},{"name":"cluster.xds_cluster.membership_healthy","value":14},{"name":"cluster.xds_cluster.membership_total","value":14},{"name":"cluster.xds_c luster.original_dst_host_invalid","value":0},{"name":"cluster.xds_cluster.retry_or_shadow_abandoned","value":0},{"name":"cluster.xds_cluster.ssl.ciphers.ECDHE-RSA-AES128-GCM-SHA256", "value":1},{"name":"cluster.xds_cluster.ssl.connection_error","value":0},{"name":"cluster.xds_cluster.ssl.curves.X25519","value":1},{"name":"cluster.xds_cluster.ssl.fail_verify_cert hash","value":0},{"name":"cluster.xds_cluster.ssl.fail_verify_error","value":0},{"name":"cluster.xds_cluster.ssl.fail_verify_no_cert","value":0},{"name":"cluster.xds_cluster.ssl.fail verify_san","value":0},{"name":"cluster.xds_cluster.ssl.handshake","value":1},{"name":"cluster.xds_cluster.ssl.no_certificate","value":0},{"name":"cluster.xds_cluster.ssl.ocsp_stapl e_failed","value":0},{"name":"cluster.xds_cluster.ssl.ocsp_staple_omitted","value":0},{"name":"cluster.xds_cluster.ssl.ocsp_staple_requests","value":0},{"name":"cluster.xds_cluster.s sl.ocsp_staple_responses","value":0},{"name":"cluster.xds_cluster.ssl.session_reused","value":0},{"name":"cluster.xds_cluster.ssl.sigalgs.rsa_pss_rsae_sha256","value":1},{"name":"clu ster.xds_cluster.ssl.versions.TLSv1.2","value":1},{"name":"cluster.xds_cluster.update_attempt","value":142165},{"name":"cluster.xds_cluster.update_empty","value":0},{"name":"cluster. xds_cluster.update_failure","value":0},{"name":"cluster.xds_cluster.update_no_rebuild","value":141952},{"name":"cluster.xds_cluster.update_success","value":142165},{"name":"cluster.x ds_cluster.upstream_cx_active","value":1},{"name":"cluster.xds_cluster.upstream_cx_close_notify","value":0},{"name":"cluster.xds_cluster.upstream_cx_connect_attempts_exceeded","value ":0},{"name":"cluster.xds_cluster.upstream_cx_connect_fail","value":0},{"name":"cluster.xds_cluster.upstream_cx_connect_timeout","value":0},{"name":"cluster.xds_cluster.upstream_cx_c onnect_with_0_rtt","value":0},{"name":"cluster.xds_cluster.upstream_cx_destroy","value":0},{"name":"cluster.xds_cluster.upstream_cx_destroy_local","value":0},{"name":"cluster.xds_clu ster.upstream_cx_destroy_local_with_active_rq","value":0},{"name":"cluster.xds_cluster.upstream_cx_destroy_remote","value":0},{"name":"cluster.xds_cluster.upstream_cx_destroy_remote with_active_rq","value":0},{"name":"cluster.xds_cluster.upstream_cx_destroy_with_active_rq","value":0},{"name":"cluster.xds_cluster.upstream_cx_http1_total","value":0},{"name":"clust er.xds_cluster.upstream_cx_http2_total","value":1},{"name":"cluster.xds_cluster.upstream_cx_http3_total","value":0},{"name":"cluster.xds_cluster.upstream_cx_idle_timeout","value":0}, {"name":"cluster.xds_cluster.upstream_cx_max_duration_reached","value":0},{"name":"cluster.xds_cluster.upstream_cx_max_requests","value":0},{"name":"cluster.xds_cluster.upstream_cx_n one_healthy","value":0},{"name":"cluster.xds_cluster.upstream_cx_overflow","value":0},{"name":"cluster.xds_cluster.upstream_cx_pool_overflow","value":0},{"name":"cluster.xds_cluster. upstream_cx_protocol_error","value":0},{"name":"cluster.xds_cluster.upstream_cx_rx_bytes_buffered","value":17},{"name":"cluster.xds_cluster.upstream_cx_rx_bytes_total","value":1690}, {"name":"cluster.xds_cluster.upstream_cx_total","value":1},{"name":"cluster.xds_cluster.upstream_cx_tx_bytes_buffered","value":0},{"name":"cluster.xds_cluster.upstream_cx_tx_bytes_to tal","value":1896},{"name":"cluster.xds_cluster.upstream_flow_control_backed_up_total","value":0},{"name":"cluster.xds_cluster.upstream_flow_control_drained_total","value":0},{"name" :"cluster.xds_cluster.upstream_flow_control_paused_reading_total","value":0},{"name":"cluster.xds_cluster.upstream_flow_control_resumed_reading_total","value":0},{"name":"cluster.xds cluster.upstream_http3_broken","value":0},{"name":"cluster.xds_cluster.upstream_internal_redirect_failed_total","value":0},{"name":"cluster.xds_cluster.upstream_internal_redirect_su cceeded_total","value":0},{"name":"cluster.xds_cluster.upstream_rq_0rtt","value":0},{"name":"cluster.xds_cluster.upstream_rq_active","value":1},{"name":"cluster.xds_cluster.upstream rq_cancelled","value":0},{"name":"cluster.xds_cluster.upstream_rq_completed","value":0},{"name":"cluster.xds_cluster.upstream_rq_maintenance_mode","value":0},{"name":"cluster.xds_clu ster.upstream_rq_max_duration_reached","value":0},{"name":"cluster.xds_cluster.upstream_rq_pending_active","value":0},{"name":"cluster.xds_cluster.upstream_rq_pending_failure_eject", "value":0},{"name":"cluster.xds_cluster.upstream_rq_pending_overflow","value":0},{"name":"cluster.xds_cluster.upstream_rq_pending_total","value":1},{"name":"cluster.xds_cluster.upstr eam_rq_per_try_idle_timeout","value":0},{"name":"cluster.xds_cluster.upstream_rq_per_try_timeout","value":0},{"name":"cluster.xds_cluster.upstream_rq_retry","value":0},{"name":"clust er.xds_cluster.upstream_rq_retry_backoff_exponential","value":0},{"name":"cluster.xds_cluster.upstream_rq_retry_backoff_ratelimited","value":0},{"name":"cluster.xds_cluster.upstream rq_retry_limit_exceeded","value":0},{"name":"cluster.xds_cluster.upstream_rq_retry_overflow","value":0},{"name":"cluster.xds_cluster.upstream_rq_retry_success","value":0},{"name":"cl uster.xds_cluster.upstream_rq_rx_reset","value":0},{"name":"cluster.xds_cluster.upstream_rq_timeout","value":0},{"name":"cluster.xds_cluster.upstream_rq_total","value":1},{"name":"cl uster.xds_cluster.upstream_rq_tx_reset","value":0},{"name":"cluster.xds_cluster.version","value":0},{"name":"cluster_manager.active_clusters","value":3},{"name":"cluster_manager.cds. init_fetch_timeout","value":0},{"name":"cluster_manager.cds.update_attempt","value":1},{"name":"cluster_manager.cds.update_failure","value":0},{"name":"cluster_manager.cds.update_rej ected","value":0},{"name":"cluster_manager.cds.update_success","value":1},{"name":"cluster_manager.cds.update_time","value":1727989634530},{"name":"cluster_manager.cds.version","valu e":17241709254077376921},{"name":"cluster_manager.cluster_added","value":3},{"name":"cluster_manager.cluster_modified","value":0},{"name":"cluster_manager.cluster_removed","value":0} ,{"name":"cluster_manager.cluster_updated","value":212},{"name":"cluster_manager.cluster_updated_via_merge","value":0},{"name":"cluster_manager.odcds.init_fetch_timeout","value":0},{ "name":"cluster_manager.odcds.update_attempt","value":0},{"name":"cluster_manager.odcds.update_failure","value":0},{"name":"cluster_manager.odcds.update_rejected","value":0},{"name": "cluster_manager.odcds.update_success","value":0},{"name":"cluster_manager.odcds.update_time","value":0},{"name":"cluster_manager.odcds.version","value":0},{"name":"cluster_manager.u pdate_merge_cancelled","value":0},{"name":"cluster_manager.update_out_of_merge_window","value":0},{"name":"cluster_manager.warming_clusters","value":0},{"name":"control_plane.connect ed_state","value":1},{"name":"control_plane.pending_requests","value":0},{"name":"control_plane.rate_limit_enforced","value":0},{"name":"dns.cares.get_addr_failure","value":0},{"name":"dns.cares.not_found","value":0},{"name":"dns.cares.pending_resolutions","value":0},{"name":"dns.cares.resolve_total","value":142165},{"name":"dns.cares.timeouts","value":0},{"name ":"envoy.overload_actions.reset_high_memory_stream.count","value":0},{"name":"filesystem.flushed_by_timer","value":71105},{"name":"filesystem.reopen_failed","value":0},{"name":"files ystem.write_buffered","value":1337951},{"name":"filesystem.write_completed","value":55153},{"name":"filesystem.write_failed","value":0},{"name":"filesystem.write_total_buffered","val ue":0},{"name":"http.admin.downstream_cx_active","value":2},{"name":"http.admin.downstream_cx_delayed_close_timeout","value":0},{"name":"http.admin.downstream_cx_destroy","value":9}, {"name":"http.admin.downstream_cx_destroy_active_rq","value":0},{"name":"http.admin.downstream_cx_destroy_local","value":0},{"name":"http.admin.downstream_cx_destroy_local_active_rq" ,"value":0},{"name":"http.admin.downstream_cx_destroy_remote","value":9},{"name":"http.admin.downstream_cx_destroy_remote_active_rq","value":0},{"name":"http.admin.downstream_cx_drai n_close","value":0},{"name":"http.admin.downstream_cx_http1_active","value":2},{"name":"http.admin.downstream_cx_http1_total","value":11},{"name":"http.admin.downstream_cx_http2_acti ve","value":0},{"name":"http.admin.downstream_cx_http2_total","value":0},{"name":"http.admin.downstream_cx_http3_active","value":0},{"name":"http.admin.downstream_cx_http3_total","va lue":0},{"name":"http.admin.downstream_cx_idle_timeout","value":0},{"name":"http.admin.downstream_cx_max_duration_reached","value":0},{"name":"http.admin.downstream_cx_max_requests_r eached","value":0},{"name":"http.admin.downstream_cx_overload_disable_keepalive","value":0},{"name":"http.admin.downstream_cx_protocol_error","value":0},{"name":"http.admin.downstrea m_cx_rx_bytes_buffered","value":196},{"name":"http.admin.downstream_cx_rx_bytes_total","value":1268965},{"name":"http.admin.downstream_cx_ssl_active","value":0},{"name":"http.admin.d ownstream_cx_ssl_total","value":0},{"name":"http.admin.downstream_cx_total","value":11},{"name":"http.admin.downstream_cx_tx_bytes_buffered","value":0},{"name":"http.admin.downstream cx_tx_bytes_total","value":1108803349},{"name":"http.admin.downstream_cx_upgrades_active","value":0},{"name":"http.admin.downstream_cx_upgrades_total","value":0},{"name":"http.admin .downstream_flow_control_paused_reading_total","value":0},{"name":"http.admin.downstream_flow_control_resumed_reading_total","value":0},{"name":"http.admin.downstream_rq_1xx","value" :0},{"name":"http.admin.downstream_rq_2xx","value":11861},{"name":"http.admin.downstream_rq_3xx","value":0},{"name":"http.admin.downstream_rq_4xx","value":0},{"name":"http.admin.down stream_rq_5xx","value":0},{"name":"http.admin.downstream_rq_active","value":1},{"name":"http.admin.downstream_rq_completed","value":11861},{"name":"http.admin.downstream_rq_failed_pa th_normalization","value":0},{"name":"http.admin.downstream_rq_header_timeout","value":0},{"name":"http.admin.downstream_rq_http1_total","value":11861},{"name":"http.admin.downstream rq_http2_total","value":0},{"name":"http.admin.downstream_rq_http3_total","value":0},{"name":"http.admin.downstream_rq_idle_timeout","value":0},{"name":"http.admin.downstream_rq_max duration_reached","value":0},{"name":"http.admin.downstream_rq_non_relative_path","value":0},{"name":"http.admin.downstream_rq_overload_close","value":0},{"name":"http.admin.downstr eam_rq_redirected_with_normalized_path","value":0},{"name":"http.admin.downstream_rq_rejected_via_ip_detection","value":0},{"name":"http.admin.downstream_rq_response_before_rq_comple te","value":0},{"name":"http.admin.downstream_rq_rx_reset","value":0},{"name":"http.admin.downstream_rq_timeout","value":0},{"name":"http.admin.downstream_rq_too_large","value":0},{" name":"http.admin.downstream_rq_total","value":11861},{"name":"http.admin.downstream_rq_tx_reset","value":0},{"name":"http.admin.downstream_rq_ws_on_non_ws_route","value":0},{"name": "http.admin.rs_too_large","value":0},{"name":"http.async-client.no_cluster","value":0},{"name":"http.async-client.no_route","value":0},{"name":"http.async-client.passthrough_internal redirect_bad_location","value":0},{"name":"http.async-client.passthrough_internal_redirect_no_route","value":0},{"name":"http.async-client.passthrough_internal_redirect_predicate"," value":0},{"name":"http.async-client.passthrough_internal_redirect_too_many_redirects","value":0},{"name":"http.async-client.passthrough_internal_redirect_unsafe_scheme","value":0},{ "name":"http.async-client.rq_direct_response","value":0},{"name":"http.async-client.rq_redirect","value":0},{"name":"http.async-client.rq_reset_after_downstream_response_started","va lue":0},{"name":"http.async-client.rq_total","value":1},{"name":"http.inbound_http.downstream_cx_active","value":94},{"name":"http.inbound_http.downstream_cx_delayed_close_timeout"," value":29},{"name":"http.inbound_http.downstream_cx_destroy","value":4035198},{"name":"http.inbound_http.downstream_cx_destroy_active_rq","value":47},{"name":"http.inbound_http.downs tream_cx_destroy_local","value":29},{"name":"http.inbound_http.downstream_cx_destroy_local_active_rq","value":0},{"name":"http.inbound_http.downstream_cx_destroy_remote","value":4035 169},{"name":"http.inbound_http.downstream_cx_destroy_remote_active_rq","value":47},{"name":"http.inbound_http.downstream_cx_drain_close","value":0},{"name":"http.inbound_http.downst ream_cx_http1_active","value":0},{"name":"http.inbound_http.downstream_cx_http1_total","value":9},{"name":"http.inbound_http.downstream_cx_http2_active","value":94},{"name":"http.inb ound_http.downstream_cx_http2_total","value":3920069},{"name":"http.inbound_http.downstream_cx_http3_active","value":0},{"name":"http.inbound_http.downstream_cx_http3_total","value": 0},{"name":"http.inbound_http.downstream_cx_idle_timeout","value":29},{"name":"http.inbound_http.downstream_cx_max_duration_reached","value":0},{"name":"http.inbound_http.downstream cx_max_requests_reached","value":0},{"name":"http.inbound_http.downstream_cx_overload_disable_keepalive","value":0},{"name":"http.inbound_http.downstream_cx_protocol_error","value":0 },{"name":"http.inbound_http.downstream_cx_rx_bytes_buffered","value":1013},{"name":"http.inbound_http.downstream_cx_rx_bytes_total","value":1592114121},{"name":"http.inbound_http.do wnstream_cx_ssl_active","value":94},{"name":"http.inbound_http.downstream_cx_ssl_total","value":4035292},{"name":"http.inbound_http.downstream_cx_total","value":4035292},{"name":"htt p.inbound_http.downstream_cx_tx_bytes_buffered","value":0},{"name":"http.inbound_http.downstream_cx_tx_bytes_total","value":734485721},{"name":"http.inbound_http.downstream_cx_upgrad es_active","value":0},{"name":"http.inbound_http.downstream_cx_upgrades_total","value":0},{"name":"http.inbound_http.downstream_flow_control_paused_reading_total","value":0},{"name": "http.inbound_http.downstream_flow_control_resumed_reading_total","value":0},{"name":"http.inbound_http.downstream_rq_1xx","value":0},{"name":"http.inbound_http.downstream_rq_2xx","v alue":3390624},{"name":"http.inbound_http.downstream_rq_3xx","value":0},{"name":"http.inbound_http.downstream_rq_4xx","value":0},{"name":"http.inbound_http.downstream_rq_5xx","value" :19},{"name":"http.inbound_http.downstream_rq_active","value":0},{"name":"http.inbound_http.downstream_rq_completed","value":3390643},{"name":"http.inbound_http.downstream_rq_failed path_normalization","value":0},{"name":"http.inbound_http.downstream_rq_header_timeout","value":0},{"name":"http.inbound_http.downstream_rq_http1_total","value":19},{"name":"http.inb ound_http.downstream_rq_http2_total","value":3391864},{"name":"http.inbound_http.downstream_rq_http3_total","value":0},{"name":"http.inbound_http.downstream_rq_idle_timeout","value": 0},{"name":"http.inbound_http.downstream_rq_max_duration_reached","value":409},{"name":"http.inbound_http.downstream_rq_non_relative_path","value":0},{"name":"http.inbound_http.downstream_rq_overload_close","value":0},{"name":"http.inbound_http.downstream_rq_redirected_with_normalized_path","value":0},{"name":"http.inbound_http.downstream_rq_rejected_via_ip_dete ction","value":0},{"name":"http.inbound_http.downstream_rq_response_before_rq_complete","value":0},{"name":"http.inbound_http.downstream_rq_rx_reset","value":1240},{"name":"http.inbo und_http.downstream_rq_timeout","value":0},{"name":"http.inbound_http.downstream_rq_too_large","value":0},{"name":"http.inbound_http.downstream_rq_total","value":3391883},{"name":"ht tp.inbound_http.downstream_rq_tx_reset","value":0},{"name":"http.inbound_http.downstream_rq_ws_on_non_ws_route","value":0},{"name":"http.inbound_http.no_cluster","value":0},{"name":" http.inbound_http.no_route","value":0},{"name":"http.inbound_http.passthrough_internal_redirect_bad_location","value":0},{"name":"http.inbound_http.passthrough_internal_redirect_no_r oute","value":0},{"name":"http.inbound_http.passthrough_internal_redirect_predicate","value":0},{"name":"http.inbound_http.passthrough_internal_redirect_too_many_redirects","value":0 },{"name":"http.inbound_http.passthrough_internal_redirect_unsafe_scheme","value":0},{"name":"http.inbound_http.rq_direct_response","value":0},{"name":"http.inbound_http.rq_redirect" ,"value":0},{"name":"http.inbound_http.rq_reset_after_downstream_response_started","value":0},{"name":"http.inbound_http.rq_total","value":3391883},{"name":"http.inbound_http.rs_too large","value":0},{"name":"http.inbound_http.tracing.client_enabled","value":0},{"name":"http.inbound_http.tracing.health_check","value":0},{"name":"http.inbound_http.tracing.not_tra ceable","value":0},{"name":"http.inbound_http.tracing.random_sampling","value":0},{"name":"http.inbound_http.tracing.service_forced","value":0},{"name":"http.outbound_http.downstream cx_active","value":0},{"name":"http.outbound_http.downstream_cx_delayed_close_timeout","value":0},{"name":"http.outbound_http.downstream_cx_destroy","value":0},{"name":"http.outboun d_http.downstream_cx_destroy_active_rq","value":0},{"name":"http.outbound_http.downstream_cx_destroy_local","value":0},{"name":"http.outbound_http.downstream_cx_destroy_local_active rq","value":0},{"name":"http.outbound_http.downstream_cx_destroy_remote","value":0},{"name":"http.outbound_http.downstream_cx_destroy_remote_active_rq","value":0},{"name":"http.outbo und_http.downstream_cx_drain_close","value":0},{"name":"http.outbound_http.downstream_cx_http1_active","value":0},{"name":"http.outbound_http.downstream_cx_http1_total","value":0},{" name":"http.outbound_http.downstream_cx_http2_active","value":0},{"name":"http.outbound_http.downstream_cx_http2_total","value":0},{"name":"http.outbound_http.downstream_cx_http3_act ive","value":0},{"name":"http.outbound_http.downstream_cx_http3_total","value":0},{"name":"http.outbound_http.downstream_cx_idle_timeout","value":0},{"name":"http.outbound_http.downs tream_cx_max_duration_reached","value":0},{"name":"http.outbound_http.downstream_cx_max_requests_reached","value":0},{"name":"http.outbound_http.downstream_cx_overload_disable_keepal ive","value":0},{"name":"http.outbound_http.downstream_cx_protocol_error","value":0},{"name":"http.outbound_http.downstream_cx_rx_bytes_buffered","value":0},{"name":"http.outbound_ht tp.downstream_cx_rx_bytes_total","value":0},{"name":"http.outbound_http.downstream_cx_ssl_active","value":0},{"name":"http.outbound_http.downstream_cx_ssl_total","value":0},{"name":" http.outbound_http.downstream_cx_total","value":0},{"name":"http.outbound_http.downstream_cx_tx_bytes_buffered","value":0},{"name":"http.outbound_http.downstream_cx_tx_bytes_total"," value":0},{"name":"http.outbound_http.downstream_cx_upgrades_active","value":0},{"name":"http.outbound_http.downstream_cx_upgrades_total","value":0},{"name":"http.outbound_http.downs tream_flow_control_paused_reading_total","value":0},{"name":"http.outbound_http.downstream_flow_control_resumed_reading_total","value":0},{"name":"http.outbound_http.downstream_rq_1x x","value":0},{"name":"http.outbound_http.downstream_rq_2xx","value":0},{"name":"http.outbound_http.downstream_rq_3xx","value":0},{"name":"http.outbound_http.downstream_rq_4xx","valu e":0},{"name":"http.outbound_http.downstream_rq_5xx","value":0},{"name":"http.outbound_http.downstream_rq_active","value":0},{"name":"http.outbound_http.downstream_rq_completed","val ue":0},{"name":"http.outbound_http.downstream_rq_failed_path_normalization","value":0},{"name":"http.outbound_http.downstream_rq_header_timeout","value":0},{"name":"http.outbound_htt p.downstream_rq_http1_total","value":0},{"name":"http.outbound_http.downstream_rq_http2_total","value":0},{"name":"http.outbound_http.downstream_rq_http3_total","value":0},{"name":"h ttp.outbound_http.downstream_rq_idle_timeout","value":0},{"name":"http.outbound_http.downstream_rq_max_duration_reached","value":0},{"name":"http.outbound_http.downstream_rq_non_rela tive_path","value":0},{"name":"http.outbound_http.downstream_rq_overload_close","value":0},{"name":"http.outbound_http.downstream_rq_redirected_with_normalized_path","value":0},{"nam e":"http.outbound_http.downstream_rq_rejected_via_ip_detection","value":0},{"name":"http.outbound_http.downstream_rq_response_before_rq_complete","value":0},{"name":"http.outbound_ht tp.downstream_rq_rx_reset","value":0},{"name":"http.outbound_http.downstream_rq_timeout","value":0},{"name":"http.outbound_http.downstream_rq_too_large","value":0},{"name":"http.outb ound_http.downstream_rq_total","value":0},{"name":"http.outbound_http.downstream_rq_tx_reset","value":0},{"name":"http.outbound_http.downstream_rq_ws_on_non_ws_route","value":0},{"na me":"http.outbound_http.no_cluster","value":0},{"name":"http.outbound_http.no_route","value":0},{"name":"http.outbound_http.passthrough_internal_redirect_bad_location","value":0},{"n ame":"http.outbound_http.passthrough_internal_redirect_no_route","value":0},{"name":"http.outbound_http.passthrough_internal_redirect_predicate","value":0},{"name":"http.outbound_htt p.passthrough_internal_redirect_too_many_redirects","value":0},{"name":"http.outbound_http.passthrough_internal_redirect_unsafe_scheme","value":0},{"name":"http.outbound_http.rds.glo bal.config_reload","value":1},{"name":"http.outbound_http.rds.global.config_reload_time_ms","value":1727989634512},{"name":"http.outbound_http.rds.global.init_fetch_timeout","value": 0},{"name":"http.outbound_http.rds.global.update_attempt","value":1},{"name":"http.outbound_http.rds.global.update_empty","value":0},{"name":"http.outbound_http.rds.global.update_fai lure","value":0},{"name":"http.outbound_http.rds.global.update_rejected","value":0},{"name":"http.outbound_http.rds.global.update_success","value":1},{"name":"http.outbound_http.rds. global.update_time","value":1727989634512},{"name":"http.outbound_http.rds.global.version","value":7148434200721666028},{"name":"http.outbound_http.rds.vhds.global.config_reload","va lue":0},{"name":"http.outbound_http.rds.vhds.global.init_fetch_timeout","value":1},{"name":"http.outbound_http.rds.vhds.global.update_attempt","value":2},{"name":"http.outbound_http. rds.vhds.global.update_empty","value":0},{"name":"http.outbound_http.rds.vhds.global.update_failure","value":0},{"name":"http.outbound_http.rds.vhds.global.update_rejected","value":0 },{"name":"http.outbound_http.rds.vhds.global.update_success","value":0},{"name":"http.outbound_http.rds.vhds.global.update_time","value":0},{"name":"http.outbound_http.rds.vhds.glob al.version","value":0},{"name":"http.outbound_http.rq_direct_response","value":0},{"name":"http.outbound_http.rq_redirect","value":0},{"name":"http.outbound_http.rq_reset_after_downs tream_response_started","value":0},{"name":"http.outbound_http.rq_total","value":0},{"name":"http.outbound_http.rs_too_large","value":0},{"name":"http.outbound_http.tracing.client_en abled","value":0},{"name":"http.outbound_http.tracing.health_check","value":0},{"name":"http.outbound_http.tracing.not_traceable","value":0},{"name":"http.outbound_http.tracing.random_sampling","value":0},{"name":"http.outbound_http.tracing.service_forced","value":0},{"name":"http1.dropped_headers_with_underscores","value":0},{"name":"http1.metadata_not_supporte d_error","value":0},{"name":"http1.requests_rejected_with_underscores_in_headers","value":0},{"name":"http1.response_flood","value":0},{"name":"http2.deferred_stream_close","value":0 },{"name":"http2.dropped_headers_with_underscores","value":0},{"name":"http2.header_overflow","value":0},{"name":"http2.headers_cb_no_stream","value":0},{"name":"http2.inbound_empty frames_flood","value":0},{"name":"http2.inbound_priority_frames_flood","value":0},{"name":"http2.inbound_window_update_frames_flood","value":0},{"name":"http2.keepalive_timeout","val ue":0},{"name":"http2.metadata_empty_frames","value":0},{"name":"http2.outbound_control_flood","value":0},{"name":"http2.outbound_flood","value":0},{"name":"http2.pending_send_bytes" ,"value":0},{"name":"http2.requests_rejected_with_underscores_in_headers","value":0},{"name":"http2.rx_messaging_error","value":0},{"name":"http2.rx_reset","value":1193},{"name":"htt p2.stream_refused_errors","value":0},{"name":"http2.streams_active","value":0},{"name":"http2.trailers","value":0},{"name":"http2.tx_flush_timeout","value":0},{"name":"http2.tx_reset ","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.downstream_cx_active","value":0},{"name":"listener./export/content/lid/a pps/code-health-policy-engine-grpc/i001/var/outbound.sock.downstream_cx_destroy","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outboun d.sock.downstream_cx_overflow","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.downstream_cx_overload_reject","value":0},{ "name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health -policy-engine-grpc/i001/var/outbound.sock.downstream_cx_transport_socket_connect_timeout","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/v ar/outbound.sock.downstream_global_cx_overflow","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.downstream_listener_filter _error","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.downstream_listener_filter_remote_close","value":0},{"name":"liste ner./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.downstream_pre_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy- engine-grpc/i001/var/outbound.sock.downstream_pre_cx_timeout","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.extension_co nfig_missing","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.http.outbound_http.downstream_rq_1xx","value":0},{"name":"li stener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.http.outbound_http.downstream_rq_2xx","value":0},{"name":"listener./export/content/lid/apps/code -health-policy-engine-grpc/i001/var/outbound.sock.http.outbound_http.downstream_rq_3xx","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/ outbound.sock.http.outbound_http.downstream_rq_4xx","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.http.outbound_http.dow nstream_rq_5xx","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.http.outbound_http.downstream_rq_completed","value":0},{"n ame":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.no_filter_chain_match","value":0},{"name":"listener./export/content/lid/apps/code-health -policy-engine-grpc/i001/var/outbound.sock.worker_0.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock. worker_0.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_1.downstream_cx_active","value":0},{" name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_1.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/cod e-health-policy-engine-grpc/i001/var/outbound.sock.worker_10.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbo und.sock.worker_10.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_11.downstream_cx_active","v alue":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_11.downstream_cx_total","value":0},{"name":"listener./export/content/ lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_12.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i0 01/var/outbound.sock.worker_12.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_13.downstream_c x_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_13.downstream_cx_total","value":0},{"name":"listener./exp ort/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_14.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-en gine-grpc/i001/var/outbound.sock.worker_14.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_15. downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_15.downstream_cx_total","value":0},{"name":"l istener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_16.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-heal th-policy-engine-grpc/i001/var/outbound.sock.worker_16.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.soc k.worker_17.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_17.downstream_cx_total","value":0 },{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_18.downstream_cx_active","value":0},{"name":"listener./export/content/lid/ap ps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_18.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/ outbound.sock.worker_19.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_19.downstream_cx_tota l","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_2.downstream_cx_active","value":0},{"name":"listener./export/con tent/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_2.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc /i001/var/outbound.sock.worker_20.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_20.downstre am_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_21.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_21.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy -engine-grpc/i001/var/outbound.sock.worker_22.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker 22.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_23.downstream_cx_active","value":0},{"name ":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_23.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-h ealth-policy-engine-grpc/i001/var/outbound.sock.worker_24.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound .sock.worker_24.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_25.downstream_cx_active","valu e":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_25.downstream_cx_total","value":0},{"name":"listener./export/content/lid /apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_26.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/ var/outbound.sock.worker_26.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_27.downstream_cx_a ctive","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_27.downstream_cx_total","value":0},{"name":"listener./export /content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_28.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engin e-grpc/i001/var/outbound.sock.worker_28.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_29.dow nstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_29.downstream_cx_total","value":0},{"name":"list ener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_3.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-p olicy-engine-grpc/i001/var/outbound.sock.worker_3.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.wor ker_30.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_30.downstream_cx_total","value":0},{"n ame":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_31.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/co de-health-policy-engine-grpc/i001/var/outbound.sock.worker_31.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbo und.sock.worker_32.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_32.downstream_cx_total","v alue":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_33.downstream_cx_active","value":0},{"name":"listener./export/content /lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_33.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i0 01/var/outbound.sock.worker_34.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_34.downstream cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_35.downstream_cx_active","value":0},{"name":"listener./ex port/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_35.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-en gine-grpc/i001/var/outbound.sock.worker_36.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_36 .downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_37.downstream_cx_active","value":0},{"name":" listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_37.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-heal th-policy-engine-grpc/i001/var/outbound.sock.worker_38.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.so ck.worker_38.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_39.downstream_cx_active","value": 0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_39.downstream_cx_total","value":0},{"name":"listener./export/content/lid/ap ps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_4.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/ outbound.sock.worker_4.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_5.downstream_cx_active" ,"value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_5.downstream_cx_total","value":0},{"name":"listener./export/conten t/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_6.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i 001/var/outbound.sock.worker_6.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_7.downstream_cx _active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_7.downstream_cx_total","value":0},{"name":"listener./expor t/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_8.downstream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engin e-grpc/i001/var/outbound.sock.worker_8.downstream_cx_total","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_9.downs tream_cx_active","value":0},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i001/var/outbound.sock.worker_9.downstream_cx_total","value":0},{"name":"listene r.[]_21519.downstream_cx_active","value":94},{"name":"listener.[]_21519.downstream_cx_destroy","value":4035198},{"name":"listener.[]_21519.downstream_cx_overflow","value":0},{" name":"listener.[]_21519.downstream_cx_overload_reject","value":0},{"name":"listener.[]_21519.downstream_cx_total","value":4035292},{"name":"listener.[]_21519.downstream_cx_tra nsport_socket_connect_timeout","value":0},{"name":"listener.[]_21519.downstream_global_cx_overflow","value":0},{"name":"listener.[]_21519.downstream_listener_filter_error","value ":0},{"name":"listener.[]_21519.downstream_listener_filter_remote_close","value":0},{"name":"listener.[]_21519.downstream_pre_cx_active","value":0},{"name":"listener.[]_21519.d ownstream_pre_cx_timeout","value":0},{"name":"listener.[]_21519.extension_config_missing","value":0},{"name":"listener.[]_21519.http.inbound_http.downstream_rq_1xx","value":0},{"name":"listener.[]_21519.http.inbound_http.downstream_rq_2xx","value":3390624},{"name":"listener.[]_21519.http.inbound_http.downstream_rq_3xx","value":0},{"name":"listener.[]_2 1519.http.inbound_http.downstream_rq_4xx","value":0},{"name":"listener.[]_21519.http.inbound_http.downstream_rq_5xx","value":19},{"name":"listener.[]_21519.http.inbound_http.down stream_rq_completed","value":3390643},{"name":"listener.[]_21519.no_filter_chain_match","value":0},{"name":"listener.[]_21519.server_ssl_socket_factory.downstream_context_secrets _not_ready","value":0},{"name":"listener.[]_21519.server_ssl_socket_factory.ssl_context_update_by_sds","value":2},{"name":"listener.[]_21519.server_ssl_socket_factory.upstream_co ntext_secrets_not_ready","value":0},{"name":"listener.[]_21519.ssl.ciphers.AES128-GCM-SHA256","value":9},{"name":"listener.[]_21519.ssl.ciphers.TLS_AES_128_GCM_SHA256","value":38 93037},{"name":"listener.[]_21519.ssl.ciphers.TLS_CHACHA20_POLY1305_SHA256","value":27040},{"name":"listener.[]_21519.ssl.connection_error","value":2373},{"name":"listener.[]_2 1519.ssl.curves.P-256","value":3450318},{"name":"listener.[]_21519.ssl.curves.X25519","value":469759},{"name":"listener.[]_21519.ssl.fail_verify_cert_hash","value":0},{"name":"li stener.[]_21519.ssl.fail_verify_error","value":0},{"name":"listener.[]_21519.ssl.fail_verify_no_cert","value":0},{"name":"listener.[]_21519.ssl.fail_verify_san","value":0},{"na me":"listener.[]_21519.ssl.handshake","value":3920086},{"name":"listener.[]_21519.ssl.no_certificate","value":1},{"name":"listener.[]_21519.ssl.ocsp_staple_failed","value":0},{ "name":"listener.[]_21519.ssl.ocsp_staple_omitted","value":469747},{"name":"listener.[]_21519.ssl.ocsp_staple_requests","value":469747},{"name":"listener.[]_21519.ssl.ocsp_stap le_responses","value":0},{"name":"listener.[]_21519.ssl.session_reused","value":2},{"name":"listener.[]_21519.ssl.sigalgs.rsa_pss_rsae_sha256","value":3920085},{"name":"listener. []_21519.ssl.versions.TLSv1.2","value":9},{"name":"listener.[]_21519.ssl.versions.TLSv1.3","value":3920077},{"name":"listener.[]_21519.worker_0.downstream_cx_active","value":1} ,{"name":"listener.[]_21519.worker_0.downstream_cx_total","value":100143},{"name":"listener.[]_21519.worker_1.downstream_cx_active","value":4},{"name":"listener.[]_21519.worker _1.downstream_cx_total","value":99416},{"name":"listener.[]_21519.worker_10.downstream_cx_active","value":1},{"name":"listener.[]_21519.worker_10.downstream_cx_total","value":103 830},{"name":"listener.[]_21519.worker_11.downstream_cx_active","value":3},{"name":"listener.[]_21519.worker_11.downstream_cx_total","value":99421},{"name":"listener.[]_21519.w orker_12.downstream_cx_active","value":1},{"name":"listener.[]_21519.worker_12.downstream_cx_total","value":97799},{"name":"listener.[]_21519.worker_13.downstream_cx_active","val ue":1},{"name":"listener.[]_21519.worker_13.downstream_cx_total","value":97907},{"name":"listener.[]_21519.worker_14.downstream_cx_active","value":2},{"name":"listener.[]_21519 .worker_14.downstream_cx_total","value":99015},{"name":"listener.[]_21519.worker_15.downstream_cx_active","value":4},{"name":"listener.[]21519.worker_15.downstream_cx_total","va lue":101220},{"name":"listener.[]_21519.worker_16.downstream_cx_active","value":2},{"name":"listener.[]21519.worker_16.downstream_cx_total","value":100551},{"name":"listener.[ ]_21519.worker_17.downstream_cx_active","value":1},{"name":"listener.[]_21519.worker_17.downstream_cx_total","value":105491},{"name":"listener.[]_21519.worker_18.downstream_cx_ac tive","value":2},{"name":"listener.[]_21519.worker_18.downstream_cx_total","value":103244},{"name":"listener.[]_21519.worker_19.downstream_cx_active","value":2},{"name":"listener .[]_21519.worker_19.downstream_cx_total","value":100443},{"name":"listener.[]_21519.worker_2.downstream_cx_active","value":3},{"name":"listener.[]21519.worker_2.downstream_cx total","value":101192},{"name":"listener.[]_21519.worker_20.downstream_cx_active","value":1},{"name":"listener.[]_21519.worker_20.downstream_cx_total","value":101752},{"name":"li stener.[]_21519.worker_21.downstream_cx_active","value":1},{"name":"listener.[]_21519.worker_21.downstream_cx_total","value":102284},{"name":"listener.[]_21519.worker_22.downst ream_cx_active","value":3},{"name":"listener.[]_21519.worker_22.downstream_cx_total","value":100451},{"name":"listener.[]_21519.worker_23.downstream_cx_active","value":4},{"name" :"listener.[]_21519.worker_23.downstream_cx_total","value":91812},{"name":"listener.[]_21519.worker_24.downstream_cx_active","value":2},{"name":"listener.[]_21519.worker_24.dow nstream_cx_total","value":99367},{"name":"listener.[]_21519.worker_25.downstream_cx_active","value":3},{"name":"listener.[]_21519.worker_25.downstream_cx_total","value":101129},{ "name":"listener.[]_21519.worker_26.downstream_cx_active","value":0},{"name":"listener.[]_21519.worker_26.downstream_cx_total","value":101635},{"name":"listener.[]_21519.worker _27.downstream_cx_active","value":4},{"name":"listener.[]_21519.worker_27.downstream_cx_total","value":96704},{"name":"listener.[]_21519.worker_28.downstream_cx_active","value":1 },{"name":"listener.[]_21519.worker_28.downstream_cx_total","value":102239},{"name":"listener.[]_21519.worker_29.downstream_cx_active","value":1},{"name":"listener.[]_21519.wor ker_29.downstream_cx_total","value":104795},{"name":"listener.[]_21519.worker_3.downstream_cx_active","value":5},{"name":"listener.[]_21519.worker_3.downstream_cx_total","value": 98557},{"name":"listener.[]_21519.worker_30.downstream_cx_active","value":2},{"name":"listener.[]_21519.worker_30.downstream_cx_total","value":102730},{"name":"listener.[]_2151 9.worker_31.downstream_cx_active","value":5},{"name":"listener.[]_21519.worker_31.downstream_cx_total","value":101628},{"name":"listener.[]_21519.worker_32.downstream_cx_active", "value":3},{"name":"listener.[]_21519.worker_32.downstream_cx_total","value":101987},{"name":"listener.[]21519.worker_33.downstream_cx_active","value":3},{"name":"listener.[__] 21519.worker_33.downstream_cx_total","value":100816},{"name":"listener.[]_21519.worker_34.downstream_cx_active","value":1},{"name":"listener.[]_21519.worker_34.downstream_cx_tota l","value":104651},{"name":"listener.[]_21519.worker_35.downstream_cx_active","value":1},{"name":"listener.[]_21519.worker_35.downstream_cx_total","value":104746},{"name":"listen er.[]_21519.worker_36.downstream_cx_active","value":3},{"name":"listener.[]_21519.worker_36.downstream_cx_total","value":98460},{"name":"listener.[]21519.worker_37.downstream cx_active","value":2},{"name":"listener.[]_21519.worker_37.downstream_cx_total","value":99435},{"name":"listener.[]_21519.worker_38.downstream_cx_active","value":4},{"name":"list ener.[]_21519.worker_38.downstream_cx_total","value":101932},{"name":"listener.[]_21519.worker_39.downstream_cx_active","value":1},{"name":"listener.[]_21519.worker_39.downstre am_cx_total","value":98096},{"name":"listener.[]_21519.worker_4.downstream_cx_active","value":3},{"name":"listener.[]_21519.worker_4.downstream_cx_total","value":101879},{"name": "listener.[]_21519.worker_5.downstream_cx_active","value":4},{"name":"listener.[]_21519.worker_5.downstream_cx_total","value":102304},{"name":"listener.[]_21519.worker_6.downst ream_cx_active","value":0},{"name":"listener.[]_21519.worker_6.downstream_cx_total","value":95319},{"name":"listener.[]_21519.worker_7.downstream_cx_active","value":5},{"name":"l istener.[]_21519.worker_7.downstream_cx_total","value":103668},{"name":"listener.[]_21519.worker_8.downstream_cx_active","value":2},{"name":"listener.[]_21519.worker_8.downstre am_cx_total","value":99875},{"name":"listener.[]_21519.worker_9.downstream_cx_active","value":3},{"name":"listener.[]_21519.worker_9.downstream_cx_total","value":107369},{"name":"listener.admin.downstream_cx_active","value":2},{"name":"listener.admin.downstream_cx_destroy","value":9},{"name":"listener.admin.downstream_cx_overflow","value":0},{"name":"listene r.admin.downstream_cx_overload_reject","value":0},{"name":"listener.admin.downstream_cx_total","value":11},{"name":"listener.admin.downstream_cx_transport_socket_connect_timeout","va lue":0},{"name":"listener.admin.downstream_global_cx_overflow","value":0},{"name":"listener.admin.downstream_listener_filter_error","value":0},{"name":"listener.admin.downstream_list ener_filter_remote_close","value":0},{"name":"listener.admin.downstream_pre_cx_active","value":0},{"name":"listener.admin.downstream_pre_cx_timeout","value":0},{"name":"listener.admi n.http.admin.downstream_rq_1xx","value":0},{"name":"listener.admin.http.admin.downstream_rq_2xx","value":11861},{"name":"listener.admin.http.admin.downstream_rq_3xx","value":0},{"nam e":"listener.admin.http.admin.downstream_rq_4xx","value":0},{"name":"listener.admin.http.admin.downstream_rq_5xx","value":0},{"name":"listener.admin.http.admin.downstream_rq_complete d","value":11861},{"name":"listener.admin.main_thread.downstream_cx_active","value":2},{"name":"listener.admin.main_thread.downstream_cx_total","value":11},{"name":"listener.admin.no filter_chain_match","value":0},{"name":"listener_manager.lds.init_fetch_timeout","value":0},{"name":"listener_manager.lds.update_attempt","value":1},{"name":"listener_manager.lds.up date_failure","value":0},{"name":"listener_manager.lds.update_rejected","value":0},{"name":"listener_manager.lds.update_success","value":1},{"name":"listener_manager.lds.update_time" ,"value":1727989634530},{"name":"listener_manager.lds.version","value":17241709254077376921},{"name":"listener_manager.listener_added","value":2},{"name":"listener_manager.listener_c reate_failure","value":0},{"name":"listener_manager.listener_create_success","value":80},{"name":"listener_manager.listener_in_place_updated","value":0},{"name":"listener_manager.lis tener_modified","value":0},{"name":"listener_manager.listener_removed","value":0},{"name":"listener_manager.listener_stopped","value":0},{"name":"listener_manager.total_filter_chains draining","value":0},{"name":"listener_manager.total_listeners_active","value":2},{"name":"listener_manager.total_listeners_draining","value":0},{"name":"listener_manager.total_list eners_warming","value":0},{"name":"listener_manager.workers_started","value":1},{"name":"main_thread.watchdog_mega_miss","value":0},{"name":"main_thread.watchdog_miss","value":0},{"n ame":"runtime.admin_overrides_active","value":0},{"name":"runtime.deprecated_feature_seen_since_process_start","value":0},{"name":"runtime.deprecated_feature_use","value":0},{"name": "runtime.load_error","value":0},{"name":"runtime.load_success","value":1},{"name":"runtime.num_keys","value":0},{"name":"runtime.num_layers","value":0},{"name":"runtime.override_dir exists","value":0},{"name":"runtime.override_dir_not_exists","value":1},{"name":"sds.tls_sds.init_fetch_timeout","value":0},{"name":"sds.tls_sds.key_rotation_failed","value":0},{"nam e":"sds.tls_sds.update_attempt","value":1},{"name":"sds.tls_sds.update_failure","value":0},{"name":"sds.tls_sds.update_rejected","value":0},{"name":"sds.tls_sds.update_success","valu e":1},{"name":"sds.tls_sds.update_time","value":1727989634500},{"name":"sds.tls_sds.version","value":17241709254077376921},{"name":"sds.validation_context_sds.init_fetch_timeout","va lue":0},{"name":"sds.validation_context_sds.key_rotation_failed","value":0},{"name":"sds.validation_context_sds.update_attempt","value":1},{"name":"sds.validation_context_sds.update failure","value":0},{"name":"sds.validation_context_sds.update_rejected","value":0},{"name":"sds.validation_context_sds.update_success","value":1},{"name":"sds.validation_context_sds .update_time","value":1727989634530},{"name":"sds.validation_context_sds.version","value":17241709254077376921},{"name":"server.compilation_settings.fips_mode","value":0},{"name":"se rver.concurrency","value":40},{"name":"server.days_until_first_cert_expiring","value":208},{"name":"server.debug_assertion_failures","value":0},{"name":"server.dropped_stat_flushes", "value":0},{"name":"server.dynamic_unknown_fields","value":0},{"name":"server.envoy_bug_failures","value":0},{"name":"server.hot_restart_epoch","value":0},{"name":"server.hot_restart generation","value":1},{"name":"server.live","value":1},{"name":"server.main_thread.watchdog_mega_miss","value":0},{"name":"server.main_thread.watchdog_miss","value":0},{"name":"ser ver.memory_allocated","value":27630560},{"name":"server.memory_heap_size","value":463470592},{"name":"server.memory_physical_size","value":470064894},{"name":"server.parent_connectio ns","value":0},{"name":"server.seconds_until_first_ocsp_response_expiring","value":0},{"name":"server.state","value":0},{"name":"server.static_unknown_fields","value":0},{"name":"ser ver.stats_recent_lookups","value":5519},{"name":"server.total_connections","value":93},{"name":"server.uptime","value":711065},{"name":"server.version","value":14530891},{"name":"ser ver.wip_protos","value":0},{"name":"server.worker_0.watchdog_mega_miss","value":0},{"name":"server.worker_0.watchdog_miss","value":0},{"name":"server.worker_1.watchdog_mega_miss","va lue":0},{"name":"server.worker_1.watchdog_miss","value":0},{"name":"server.worker_10.watchdog_mega_miss","value":0},{"name":"server.worker_10.watchdog_miss","value":0},{"name":"serve r.worker_11.watchdog_mega_miss","value":0},{"name":"server.worker_11.watchdog_miss","value":0},{"name":"server.worker_12.watchdog_mega_miss","value":0},{"name":"server.worker_12.watc hdog_miss","value":0},{"name":"server.worker_13.watchdog_mega_miss","value":0},{"name":"server.worker_13.watchdog_miss","value":0},{"name":"server.worker_14.watchdog_mega_miss","valu e":0},{"name":"server.worker_14.watchdog_miss","value":0},{"name":"server.worker_15.watchdog_mega_miss","value":0},{"name":"server.worker_15.watchdog_miss","value":0},{"name":"server .worker_16.watchdog_mega_miss","value":0},{"name":"server.worker_16.watchdog_miss","value":0},{"name":"server.worker_17.watchdog_mega_miss","value":0},{"name":"server.worker_17.watch dog_miss","value":0},{"name":"server.worker_18.watchdog_mega_miss","value":0},{"name":"server.worker_18.watchdog_miss","value":0},{"name":"server.worker_19.watchdog_mega_miss","value ":0},{"name":"server.worker_19.watchdog_miss","value":0},{"name":"server.worker_2.watchdog_mega_miss","value":0},{"name":"server.worker_2.watchdog_miss","value":0},{"name":"server.wo rker_20.watchdog_mega_miss","value":0},{"name":"server.worker_20.watchdog_miss","value":0},{"name":"server.worker_21.watchdog_mega_miss","value":0},{"name":"server.worker_21.watchdog miss","value":0},{"name":"server.worker_22.watchdog_mega_miss","value":0},{"name":"server.worker_22.watchdog_miss","value":0},{"name":"server.worker_23.watchdog_mega_miss","value":0 },{"name":"server.worker_23.watchdog_miss","value":0},{"name":"server.worker_24.watchdog_mega_miss","value":0},{"name":"server.worker_24.watchdog_miss","value":0},{"name":"server.wor ker_25.watchdog_mega_miss","value":0},{"name":"server.worker_25.watchdog_miss","value":0},{"name":"server.worker_26.watchdog_mega_miss","value":0},{"name":"server.worker_26.watchdog miss","value":0},{"name":"server.worker_27.watchdog_mega_miss","value":0},{"name":"server.worker_27.watchdog_miss","value":0},{"name":"server.worker_28.watchdog_mega_miss","value":0} ,{"name":"server.worker_28.watchdog_miss","value":0},{"name":"server.worker_29.watchdog_mega_miss","value":0},{"name":"server.worker_29.watchdog_miss","value":0},{"name":"server.work er_3.watchdog_mega_miss","value":0},{"name":"server.worker_3.watchdog_miss","value":0},{"name":"server.worker_30.watchdog_mega_miss","value":0},{"name":"server.worker_30.watchdog_mis s","value":0},{"name":"server.worker_31.watchdog_mega_miss","value":0},{"name":"server.worker_31.watchdog_miss","value":0},{"name":"server.worker_32.watchdog_mega_miss","value":0},{"name":"server.worker_32.watchdog_miss","value":0},{"name":"server.worker_33.watchdog_mega_miss","value":0},{"name":"server.worker_33.watchdog_miss","value":0},{"name":"server.worker 34.watchdog_mega_miss","value":0},{"name":"server.worker_34.watchdog_miss","value":0},{"name":"server.worker_35.watchdog_mega_miss","value":0},{"name":"server.worker_35.watchdog_miss ","value":0},{"name":"server.worker_36.watchdog_mega_miss","value":0},{"name":"server.worker_36.watchdog_miss","value":0},{"name":"server.worker_37.watchdog_mega_miss","value":0},{"n ame":"server.worker_37.watchdog_miss","value":0},{"name":"server.worker_38.watchdog_mega_miss","value":0},{"name":"server.worker_38.watchdog_miss","value":0},{"name":"server.worker_3 9.watchdog_mega_miss","value":0},{"name":"server.worker_39.watchdog_miss","value":0},{"name":"server.worker_4.watchdog_mega_miss","value":0},{"name":"server.worker_4.watchdog_miss"," value":0},{"name":"server.worker_5.watchdog_mega_miss","value":0},{"name":"server.worker_5.watchdog_miss","value":0},{"name":"server.worker_6.watchdog_mega_miss","value":0},{"name":" server.worker_6.watchdog_miss","value":0},{"name":"server.worker_7.watchdog_mega_miss","value":0},{"name":"server.worker_7.watchdog_miss","value":0},{"name":"server.worker_8.watchdog _mega_miss","value":0},{"name":"server.worker_8.watchdog_miss","value":0},{"name":"server.worker_9.watchdog_mega_miss","value":0},{"name":"server.worker_9.watchdog_miss","value":0},{ "name":"workers.watchdog_mega_miss","value":0},{"name":"workers.watchdog_miss","value":0},{"histograms":{"computed_quantiles":[{"name":"cluster.uds_inbound_local_app_cluster.upstream _cx_connect_ms","values":[{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"cumulative ":null,"interval":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"interval":nu ll,"cumulative":null}]},{"name":"cluster.uds_inbound_local_app_cluster.upstream_cx_length_ms","values":[{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"cumu lative":null,"interval":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"interv al":null,"cumulative":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null}]},{"name":"cluster.uds_inbound_local_app_cluster_grpc.external.upstream_rq_time"," values":[{"cumulative":0,"interval":0},{"interval":0,"cumulative":0},{"interval":0,"cumulative":0},{"cumulative":0,"interval":0},{"interval":0,"cumulative":0},{"interval":0,"cumulati ve":0},{"interval":1.057,"cumulative":17.302549019606566},{"cumulative":282.30972762645405,"interval":1.0784999999999996},{"cumulative":1956.5301775148403,"interval":1.09570000000000 01},{"interval":1.1,"cumulative":250000}]},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_connect_ms","values":[{"interval":null,"cumulative":0},{"interval":null,"cu mulative":0},{"interval":null,"cumulative":0},{"interval":null,"cumulative":0},{"cumulative":0,"interval":null},{"interval":null,"cumulative":0},{"interval":null,"cumulative":0},{"cu mulative":0,"interval":null},{"interval":null,"cumulative":0},{"cumulative":0,"interval":null}]},{"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_cx_length_ms","values":[ {"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{" cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null}]},{" values":[{"interval":0,"cumulative":0},{"cumulative":0,"interval":0},{"interval":0,"cumulative":0},{"interval":0,"cumulative":0},{"cumulative":0,"interval":0},{"interval":0,"cumulati ve":0},{"interval":1.057,"cumulative":17.302549019606566},{"interval":1.0784999999999996,"cumulative":282.30972762645405},{"cumulative":1956.5301775148403,"interval":1.09570000000000 01},{"interval":1.1,"cumulative":250000}],"name":"cluster.uds_inbound_local_app_cluster_grpc.upstream_rq_time"},{"values":[{"cumulative":0,"interval":0},{"cumulative":0,"interval":0} ,{"cumulative":0,"interval":0},{"interval":0,"cumulative":0},{"cumulative":0,"interval":0},{"cumulative":0,"interval":0},{"interval":1.057,"cumulative":17.302549019606566},{"interval ":1.0784999999999996,"cumulative":282.30972762645405},{"cumulative":1956.5301775148403,"interval":1.0957000000000001},{"cumulative":250000,"interval":1.1}],"name":"cluster.uds_inboun d_local_app_cluster_grpc.zone.corp-lca1..upstream_rq_time"},{"values":[{"cumulative":8,"interval":null},{"interval":null,"cumulative":8.025},{"interval":null,"cumulative":8.05},{"int erval":null,"cumulative":8.075},{"cumulative":8.09,"interval":null},{"interval":null,"cumulative":8.095},{"cumulative":8.099,"interval":null},{"interval":null,"cumulative":8.0995},{" interval":null,"cumulative":8.0999},{"cumulative":8.1,"interval":null}],"name":"cluster.xds_cluster.upstream_cx_connect_ms"},{"values":[{"cumulative":null,"interval":null},{"interval ":null,"cumulative":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"cumulative ":null,"interval":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null}],"name":"cluster.xds_cluster.upstream_cx_length_ms "},{"name":"cluster_manager.cds.update_duration","values":[{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":n ull,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"interval":null, "cumulative":null},{"cumulative":null,"interval":null}]},{"values":[{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"inte rval":null,"cumulative":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"interv al":null,"cumulative":null},{"interval":null,"cumulative":null}],"name":"cluster_manager.odcds.update_duration"},{"name":"http.admin.downstream_cx_length_ms","values":[{"interval":nu ll,"cumulative":0},{"cumulative":0,"interval":null},{"cumulative":0,"interval":null},{"cumulative":0,"interval":null},{"interval":null,"cumulative":4.055},{"cumulative":4.0775,"inter val":null},{"cumulative":4.0955,"interval":null},{"cumulative":4.09775,"interval":null},{"cumulative":4.09955,"interval":null},{"interval":null,"cumulative":4.1}]},{"values":[{"cumul ative":0,"interval":null},{"interval":null,"cumulative":2.0408619975134688},{"cumulative":2.0818206934659482,"interval":null},{"cumulative":3.0366119005328596,"interval":null},{"inte rval":null,"cumulative":3.0761101243339253},{"interval":null,"cumulative":3.0892761989342805},{"cumulative":3.0998090586145648,"interval":null},{"cumulative":4.0512121212121217,"inte rval":null},{"cumulative":4.0991313131313127,"interval":null},{"cumulative":5.1,"interval":null}],"name":"http.admin.downstream_rq_time"},{"name":"http.inbound_http.downstream_cx_len gth_ms","values":[{"cumulative":0,"interval":3},{"cumulative":75.238378518921664,"interval":79.333333333333329},{"cumulative":86.938280590248453,"interval":91.5},{"interval":113.3333 3333333334,"cumulative":10025.138457009329},{"cumulative":656108.8798820927,"interval":128.66666666666669},{"interval":237.99999999999994,"cumulative":1314810.3259821238},{"cumulativ e":1872020.1046524539,"interval":9878},{"interval":9889,"cumulative":1886021.8500166554},{"interval":9897.8,"cumulative":1897223.2463080168},{"cumulative":3700000,"interval":9900}]},"},{"name":"cluster_manager.cds.update_duration","values":[{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":n ull,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"interval":null, "cumulative":null},{"cumulative":null,"interval":null}]},{"values":[{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"inte rval":null,"cumulative":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"interv al":null,"cumulative":null},{"interval":null,"cumulative":null}],"name":"cluster_manager.odcds.update_duration"},{"name":"http.admin.downstream_cx_length_ms","values":[{"interval":nu ll,"cumulative":0},{"cumulative":0,"interval":null},{"cumulative":0,"interval":null},{"cumulative":0,"interval":null},{"interval":null,"cumulative":4.055},{"cumulative":4.0775,"inter val":null},{"cumulative":4.0955,"interval":null},{"cumulative":4.09775,"interval":null},{"cumulative":4.09955,"interval":null},{"interval":null,"cumulative":4.1}]},{"values":[{"cumul ative":0,"interval":null},{"interval":null,"cumulative":2.0408619975134688},{"cumulative":2.0818206934659482,"interval":null},{"cumulative":3.0366119005328596,"interval":null},{"inte rval":null,"cumulative":3.0761101243339253},{"interval":null,"cumulative":3.0892761989342805},{"cumulative":3.0998090586145648,"interval":null},{"cumulative":4.0512121212121217,"inte rval":null},{"cumulative":4.0991313131313127,"interval":null},{"cumulative":5.1,"interval":null}],"name":"http.admin.downstream_rq_time"},{"name":"http.inbound_http.downstream_cx_len gth_ms","values":[{"cumulative":0,"interval":3},{"cumulative":75.238378518921664,"interval":79.333333333333329},{"cumulative":86.938280590248453,"interval":91.5},{"interval":113.3333 3333333334,"cumulative":10025.138457009329},{"cumulative":656108.8798820927,"interval":128.66666666666669},{"interval":237.99999999999994,"cumulative":1314810.3259821238},{"cumulativ e":1872020.1046524539,"interval":9878},{"interval":9889,"cumulative":1886021.8500166554},{"interval":9897.8,"cumulative":1897223.2463080168},{"cumulative":3700000,"interval":9900}]}, {"name":"http.inbound_http.downstream_rq_time","values":[{"interval":0,"cumulative":0},{"interval":0,"cumulative":0},{"interval":0,"cumulative":1.0033930021254738},{"cumulative":1.05 55701507898354,"interval":1.043421052631579},{"interval":1.0773684210526315,"cumulative":1.0868764399884525},{"cumulative":1.0973118697213247,"interval":1.0886842105263157},{"interva l":1.0977368421052631,"cumulative":290.565760869567},{"interval":1.0988684210526316,"cumulative":669.10709219857563},{"cumulative":2034.4394160584159,"interval":1.0997736842105263},{ "interval":1.1,"cumulative":250000}]},{"values":[{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"cumulative":null,"inter val":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"interval":null,"cumulativ e":null},{"cumulative":null,"interval":null}],"name":"http.outbound_http.downstream_cx_length_ms"},{"name":"http.outbound_http.downstream_rq_time","values":[{"interval":null,"cumulat ive":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interva l":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null}]},{"name":"http.outbound_http .rds.global.update_duration","values":[{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null} ,{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{ "interval":null,"cumulative":null}]},{"name":"http.outbound_http.rds.vhds.global.update_duration","values":[{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{" cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"in terval":null,"cumulative":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null}]},{"name":"listener./export/content/lid/apps/code-health-policy-engine-grpc/i0 01/var/outbound.sock.downstream_cx_length_ms","values":[{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null ,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"cumulative":null," interval":null},{"interval":null,"cumulative":null}]},{"values":[{"interval":3,"cumulative":0},{"interval":79.333333333333329,"cumulative":75.212978688874173},{"cumulative":86.890643 039476331,"interval":91.5},{"cumulative":10025.084642844,"interval":113.33333333333334},{"interval":128.66666666666669,"cumulative":656108.8798820927},{"cumulative":1314810.325982123 8,"interval":237.99999999999994},{"interval":9878,"cumulative":1872020.1046524539},{"interval":9889,"cumulative":1886021.8500166554},{"interval":9897.8,"cumulative":1897223.246308016 8},{"cumulative":3700000,"interval":9900}],"name":"listener.[__]_21519.downstream_cx_length_ms"},{"values":[{"interval":null,"cumulative":0},{"interval":null,"cumulative":0},{"cumula tive":0,"interval":null},{"interval":null,"cumulative":0},{"interval":null,"cumulative":4.055},{"cumulative":4.0775,"interval":null},{"interval":null,"cumulative":4.0955},{"interval" :null,"cumulative":4.09775},{"cumulative":4.09955,"interval":null},{"cumulative":4.1,"interval":null}],"name":"listener.admin.downstream_cx_length_ms"},{"values":[{"interval":null,"c umulative":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"interval":null,"cum ulative":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null}],"name":"listener_manag er.lds.update_duration"},{"name":"sds.tls_sds.update_duration","values":[{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{ "cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"i nterval":null,"cumulative":null},{"cumulative":null,"interval":null}]},{"name":"sds.validation_context_sds.update_duration","values":[{"cumulative":null,"interval":null},{"cumulative ":null,"interval":null},{"cumulative":null,"interval":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null},{"cumulative":null,"interval":null},{"cumulative": null,"interval":null},{"interval":null,"cumulative":null},{"interval":null,"cumulative":null},{"cumulative":null,"interval":null}]},{"values":[{"interval":null,"cumulative":15000},{" interval":null,"cumulative":15250},{"cumulative":15500,"interval":null},{"cumulative":15750,"interval":null},{"cumulative":15900,"interval":null},{"cumulative":15950,"interval":null} ,{"interval":null,"cumulative":15990},{"interval":null,"cumulative":15995},{"cumulative":15999,"interval":null},{"cumulative":16000,"interval":null}],"name":"server.initialization_ti me_ms"}],"supported_quantiles":[0,25,50,75,90,95,99,99.5,99.9,100]}}]}

Note: If there are privacy concerns, sanitize the data prior to sharing.

Config:

Include the config used to configure Envoy.

Logs: Envoy trace logs: [2024-10-09 16:57:46.831][182][trace][connection] [source/common/network/connection_impl.cc:568] [C8] socket event: 3 [2024-10-09 16:57:46.831][182][trace][connection] [source/common/network/connection_impl.cc:679] [C8] write ready [2024-10-09 16:57:46.831][182][trace][connection] [source/common/network/connection_impl.cc:608] [C8] read ready. dispatch_buffered_data=0 [2024-10-09 16:57:46.831][182][trace][connection] [source/common/network/raw_buffer_socket.cc:24] [C8] read returns: 26 [2024-10-09 16:57:46.831][182][trace][connection] [source/common/network/raw_buffer_socket.cc:38] [C8] read error: Resource temporarily unavailable [2024-10-09 16:57:46.831][182][trace][http2] [source/common/http/http2/codec_impl.cc:969] [C8] dispatching 26 bytes [2024-10-09 16:57:46.831][182][trace][http2] [source/common/http/http2/codec_impl.cc:1071] [C8] about to recv frame type=3, flags=0, stream_id=45913 [2024-10-09 16:57:46.831][182][trace][http2] [source/common/http/http2/codec_impl.cc:2024] [C8] track inbound frame type=3 flags=0 length=4 padding_length=0 [2024-10-09 16:57:46.831][182][trace][http2] [source/common/http/http2/codec_impl.cc:1097] [C8] recv frame type=3 [2024-10-09 16:57:46.831][182][trace][http2] [source/common/http/http2/codec_impl.cc:1202] [C8] remote reset: 2 [2024-10-09 16:57:46.831][182][debug][http2] [source/common/http/http2/codec_impl.cc:1363] [C8] stream 45913 closed: 2 [2024-10-09 16:57:46.831][182][debug][client] [source/common/http/codec_client.cc:156] [C8] request reset [2024-10-09 16:57:46.831][182][trace][main] [source/common/event/dispatcher_impl.cc:250] item added to deferred deletion list (size=1) [2024-10-09 16:57:46.831][182][debug][pool] [source/common/conn_pool/conn_pool_base.cc:215] [C8] destroying stream: 0 remaining [2024-10-09 16:57:46.831][182][debug][router] [source/common/router/router.cc:1208] [C1779151][S16507185947364583566] upstream reset: reset reason: remote reset, transport failure reason: [2024-10-09 16:57:46.832][182][trace][main] [source/common/event/dispatcher_impl.cc:250] item added to deferred deletion list (size=2) [2024-10-09 16:57:46.832][182][trace][main] [source/common/event/dispatcher_impl.cc:250] item added to deferred deletion list (size=3) [2024-10-09 16:57:46.832][182][debug][http] [source/common/http/filter_manager.cc:967] [C1779151][S16507185947364583566] Sending local reply with details upstream_reset_before_response_started{remote_reset} [2024-10-09 16:57:46.832][182][debug][http] [source/common/http/conn_manager_impl.cc:1596] [C1779151][S16507185947364583566] encoding headers via codec (end_stream=true): ':status', '200' 'content-type', 'application/grpc' 'grpc-status', '14' 'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: remote reset' 'date', 'Wed, 09 Oct 2024 16:57:46 GMT' 'server', 'envoy'

Note: If there are privacy concerns, sanitize the data prior to sharing.

Call Stack: NO crash

Oct 12 '24 01:10 thinkingotter

@logstashbugreporter have you tried to reach out to gRPC folks about this?

Oct 18 '24 17:10 tyxia

@tyxia which gRPC forum I should reach out to?

Here are python gRPC debug logs:

I1018 16:08:25.517688418 44342 completion_queue.cc:1068] I1018 16:08:25.517734594 44342 completion_queue.cc:956] il)) I1018 16:08:25.629635558 44342 tcp_posix.cc:1147] I1018 16:08:25.629647979 44342 tcp_posix.cc:934] I1018 16:08:25.629658984 44342 tcp_posix.cc:869] I1018 16:08:25.629663057 44342 tcp_posix.cc:871] D1018 16:08:25.629668138 44342 tcp_posix.cc:877] a 6e 41 39 24 ad 15 62 09 d0 2c d3 32 3a b3 .' I1018 16:08:25.629690613 44342 parsing.cc:338] D1018 16:08:25.629709085 44342 chttp2_transport.cc:812] I1018 16:08:25.629721515 44342 call.cc:1458] D1018 16:08:25.629725739 44342 call.cc:1802] I1018 16:08:25.629729450 44342 parsing.cc:720] I1018 16:08:25.629732383 44342 parsing.cc:761] D1018 16:08:25.629734760 44342 parsing.cc:892] D1018 16:08:25.629742673 44342 hpack_parser.cc:706] D1018 16:08:25.629746691 44342 hpack_parser.cc:706] D1018 16:08:25.629750324 44342 hpack_parser.cc:706] D1018 16:08:25.629753773 44342 hpack_parser.cc:706] D1018 16:08:25.629770370 44342 hpack_parser.cc:706] D1018 16:08:25.629773465 44342 hpack_parser.cc:706] D1018 16:08:25.629775767 44342 hpack_parser.cc:706] D1018 16:08:25.629779159 44342 hpack_parser.cc:706] D1018 16:08:25.629782171 44342 hpack_parser.cc:706] D1018 16:08:25.629785212 44342 hpack_parser.cc:706] D1018 16:08:25.629793626 44342 hpack_parser.cc:706] D1018 16:08:25.629801776 44342 hpack_parser.cc:706] D1018 16:08:25.629817193 44342 chttp2_transport.cc:2304] D1018 16:08:25.629839861 44342 chttp2_transport.cc:2092] D1018 16:08:25.629843338 44342 chttp2_transport.cc:2092] RETURN_EVENT[0x5d6cf08258a0]: QUEUE_TIMEOUT grpc_completion_queue_next(cq=0x5d6cf08258a0, deadline=gpr_timespec { tv_sec: 1729267705, tv_nsec: 717727100, clock_type: 1 }, reserved=(n TCP:0x7c1100055b40 got_read: OK TCP:0x7c1100055b40 do_read TCP:0x7c1100055b40 call_cb 0x7c11000579a0 0x7c116b019230:0x7c1100057710 READ 0x7c1100055b40 (peer=unix:) error=OK READ DATA: 00 00 3d 01 04 00 00 09 c5 04 95 62 6b 2b 22 f3 94 74 26 75 fb 85 7c 65 1d 09 9d 8b d3 94 9d 7f c8 83 87 c2 c1 c7 c6 c5 bf 7e 9 e3 79 f5 86 e4 92 33 18 dc 7d a9 65 c4 00 00 05 00 01 00 00 09 c5 00 00 00 00 00 '..=........bk+"..t&u..|e.................~.nA9$..b..,.2:..y....3..}.e.............. INCOMING[0x7c1100057710]: HEADERS:END_HEADERS len:61 id:0x000009c5 HTTP:0x7c1100057710/0x7c11000e5e40 creating accept stream 2501 [from 0x9c5] ops[0]: RECV_INITIAL_METADATA ptr=0x7c11000e5578 BATCH:0x7c11000e6d30 START:{kRecvInitialMetadata} BATCH: RECV_INITIAL_METADATA (tag:0x7c11000e5590) [t:0x7c1100057710 fd:35 peer:unix:] Accepting new stream; num_incoming_streams_before_settings_ack=4294966045 parsing initial_metadata INCOMING[0x7c1100057710;0x7c11000e5e40]: Parse 61b last frame fragment with header HTTP:2501:HDR:SVR: :path: /grpc.health.v1.Health/Check HTTP:2501:HDR:SVR: :authority: lca1-app38344.corp.linkedin.com:21519 HTTP:2501:HDR:SVR: :method: POST HTTP:2501:HDR:SVR: :scheme: https HTTP:2501:HDR:SVR: content-type: application/grpc HTTP:2501:HDR:SVR: te: trailers HTTP:2501:HDR:SVR: grpc-accept-encoding: identity, deflate, gzip HTTP:2501:HDR:SVR: grpc-timeout: 9900ms HTTP:2501:HDR:SVR: user-agent: grpc-python/1.66.2 grpc-c/43.0.0 (linux; chttp2) HTTP:2501:HDR:SVR: x-forwarded-proto: https HTTP:2501:HDR:SVR: x-request-id: 5d26df4e-c270-43d7-9a89-a6dd3ba694ff (parse error: INTERNAL: Error parsing ':method' metadata [type.googleapis.com/grpc.status.int.stream_id='0']) HTTP:2501:HDR:SVR:x-forwarded-client-cert:By=urn:li:servicePrincipal(.....);By=urn:li:servicePrincipalMetadata(XXXX;1.0.0);Hash=XXXXX;Cert=".....-----END%20CERTIFICATE-----%0A";Subject="CN=x.y.z.biz,OU=Tools,O=XYZ,L=Mountain View,ST=California,C=US"; (parse error: INTERNAL: Error parsing ':method' metadata [type.googleapis.com/grpc.status.int.stream_id='0'])E1018 16:08:25.629810589 44342 parsing.cc:904] INCOMING[0x7c1100057710;0x7c11000e5e40]: Parse failed with INTERNAL: Error parsing ':method' metadata [type.googleapis.com/grpc.status.int.stream_id='0'] MARK_STREAM_CLOSED: t=0x7c1100057710 s=0x7c11000e5e40(id=2501) read+write [INTERNAL:Error parsing ':method' metadata {stream_id:0}] maybe_complete_recv_trailing_metadata cli=0 s=0x7c11000e5e40 closure=(nil) read_closed=1 write_closed=1 0 maybe_complete_recv_trailing_metadata cli=0 s=0x7c11000e5e40 closure=(nil) read_closed=1 write_closed=1 0

Oct 18 '24 18:10 thinkingotter

Google gRPC team who developed gRPC.

Just want to confirmed you are not using envoy gRPC client right?

Oct 18 '24 18:10 tyxia

@tyxia, filed gRPC bug https://github.com/grpc/grpc/issues/37953 This issue appears very similar to the previously reported one: https://github.com/grpc/grpc/issues/34721, which is also related to Envoy.

By the way, even though the bug seems to be in the gRPC library, it appears to be triggered specifically in conjunction with Envoy.

Oct 22 '24 05:10 thinkingotter

@tyxia what is envoy gRPC client? the client we used to send the grpc health check requests is a python client

Oct 22 '24 05:10 thinkingotter

I have linked envoy grpc in my comment above, which seems to be unrelated here.

Also the similar grpc issue you newly added seems to be unrelated to envoy. Let's use grpc issue to track and please reopen if you think it is needed

Oct 22 '24 13:10 tyxia