envoy icon indicating copy to clipboard operation
envoy copied to clipboard
verified publisher icon envoyproxy

Tap filter segmentation fault

Open nezdolik opened this issue 9 months ago • 4 comments

If you are reporting any crash or any potential security issue, do not open an issue in this repo. Please report the issue via emailing [email protected] where the issue will be triaged appropriately.

Title: Tap filter segmentation fault

Description:

When the tap filter is enabled, a segmentation fault happens when envoy receives a request with a big enough header.

Repro steps:

Attached curl command: request

Note: The Envoy_collect tool gathers a tarball with debug logs, config and the following admin endpoints: /stats, /clusters and /server_info. Please note if there are privacy concerns, sanitize the data prior to sharing the tarball/pasting.

Admin and Stats Output:

Include the admin output for the following endpoints: /stats, /clusters, /routes, /server_info. For more information, refer to the admin endpoint documentation.

Note: If there are privacy concerns, sanitize the data prior to sharing.

Config:

static_resources:

  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
          http_filters:
          
          - name: envoy.filters.http.tap
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.tap.v3.Tap
              common_config:
                static_config:
                  match_config:
                    any_match: true
                  output_config:
                    streaming: true
                    sinks:
                    - format: PROTO_BINARY_LENGTH_DELIMITED
                      file_per_tap:
                        path_prefix: taps/
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router  
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  host_rewrite_literal: www.envoyproxy.io
                  cluster: service_envoyproxy_io

  clusters:
  - name: service_envoyproxy_io
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    load_assignment:
      cluster_name: service_envoyproxy_io
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: www.envoyproxy.io
                port_value: 443
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        sni: www.envoyproxy.io

Logs:

[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:129] Caught Segmentation fault, suspect faulting address 0x0
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:113] Backtrace (use tools/stack_decode.py to get line numbers):
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:114] Envoy version: 2135e1a42f002a939d60581096291acb6abce695/1.33.2/Clean/RELEASE/BoringSSL
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:116] Address mapping: 5d25e7481000-5d25ea025000 <redacted>/envoy-1.33.2
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #0: [0x7a8b22645250]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #1: [0x5d25e7a7d313]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #2: [0x5d25e7a7f2bb]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #3: [0x5d25e9611db3]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #4: [0x5d25e986e610]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #5: [0x5d25e986e793]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #6: [0x5d25e9617090]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #7: [0x5d25e9615ec7]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #8: [0x5d25e957eeb3]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #9: [0x5d25e957ef42]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #10: [0x5d25e95788a3]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #11: [0x5d25e9577e03]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #12: [0x5d25e957e931]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #13: [0x5d25e9554dac]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #14: [0x5d25e97f7645]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #15: [0x5d25e94ec19b]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #16: [0x5d25e94e8381]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #17: [0x5d25e94f2236]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #18: [0x5d25e8f7d696]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #19: [0x5d25e8f7ebe5]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #20: [0x5d25e99f8af0]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #21: [0x5d25e99f7431]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #22: [0x5d25e8f74c8d]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #23: [0x5d25e9a7f952]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #24: [0x7a8b226a1e2e]
[2025-03-26 19:06:37.367][318026][critical][backtrace] [./source/server/backtrace.h:123] #25: [0x7a8b22733a4c]
Http1::ConnectionImpl 0x353eff7b6a10, dispatching_: 1, dispatching_slice_already_drained_: 0, reset_stream_called_: 0, handling_upgrade_: 0, deferred_end_stream_headers_: 0, processing_trailers_: 0, buffered_body_.length(): 0, header_parsing_state_: Field, current_header_field_: , current_header_value_:
active_request_:
, request_url_: /, response_encoder_.local_end_stream_: 0
absl::get<RequestHeaderMapPtr>(headers_or_trailers_):
current_dispatching_buffer_ front_slice length: 16384 contents: "111111111111111111111111111111111111111111111111111111111111111<truncated>"
ConnectionImpl 0x353eff93a190, connecting_: 0, bind_error_: 0, state(): Open, read_buffer_limit_: 1048576
socket_:
  ListenSocketImpl 0x353eff751480, transport_protocol_: raw_buffer
  connection_info_provider_:
    ConnectionInfoSetterImpl 0x353eff430788, remote_address_: [127.0.0.1:41170](http://127.0.0.1:41170/), direct_remote_address_: [127.0.0.1:41170](http://127.0.0.1:41170/), local_address_: [127.0.0.1:10000](http://127.0.0.1:10000/), server_name_:
Segmentation fault (core dumped)

Note: If there are privacy concerns, sanitize the data prior to sharing.

Call Stack:

If the Envoy binary is crashing, a call stack is required. Please refer to the Bazel Stack trace documentation.

[2025-04-01 18:07:22.384][22][critical][backtrace] [./source/server/backtrace.h:129] Caught Segmentation fault, suspect faulting address 0x0
[2025-04-01 18:07:22.384][22][critical][backtrace] [./source/server/backtrace.h:113] Backtrace (use tools/stack_decode.py to get line numbers):
[2025-04-01 18:07:22.384][22][critical][backtrace] [./source/server/backtrace.h:114] Envoy version: 76a294abce9a9d6e81934d4ff185a438b49f217a/1.34.0-dev/Clean/RELEASE/BoringSSL
[2025-04-01 18:07:22.384][22][critical][backtrace] [./source/server/backtrace.h:116] Address mapping: 5c0807cb4000-5c080a96d000 /usr/local/bin/envoy
[2025-04-01 18:07:22.384][22][critical][backtrace] [./source/server/backtrace.h:123] #0: [0x7411ae678520]
[2025-04-01 18:07:22.386][22][critical][backtrace] [./source/server/backtrace.h:121] #1: Envoy::Extensions::HttpFilters::TapFilter::HttpPerRequestTapperImpl::onResponseHeaders() [0x5c08082ca093]
[2025-04-01 18:07:22.387][22][critical][backtrace] [./source/server/backtrace.h:121] #2: Envoy::Extensions::HttpFilters::TapFilter::Filter::encodeHeaders() [0x5c08082cc22b]
[2025-04-01 18:07:22.389][22][critical][backtrace] [./source/server/backtrace.h:121] #3: Envoy::Http::FilterManager::encodeHeaders() [0x5c0809ed3012]
[2025-04-01 18:07:22.390][22][critical][backtrace] [./source/server/backtrace.h:121] #4: Envoy::Http::Utility::encodeLocalReply() [0x5c080a163b80]
[2025-04-01 18:07:22.391][22][critical][backtrace] [./source/server/backtrace.h:121] #5: Envoy::Http::Utility::sendLocalReply() [0x5c080a163d33]
[2025-04-01 18:07:22.393][22][critical][backtrace] [./source/server/backtrace.h:121] #6: Envoy::Http::DownstreamFilterManager::sendLocalReplyViaFilterChain() [0x5c0809ed83a5]
[2025-04-01 18:07:22.394][22][critical][backtrace] [./source/server/backtrace.h:121] #7: Envoy::Http::DownstreamFilterManager::sendLocalReply() [0x5c0809ed71cf]
[2025-04-01 18:07:22.395][22][critical][backtrace] [./source/server/backtrace.h:121] #8: Envoy::Http::Http1::ServerConnectionImpl::sendProtocolError() [0x5c0809e3ab7a]
[2025-04-01 18:07:22.396][22][critical][backtrace] [./source/server/backtrace.h:121] #9: Envoy::Http::Http1::ServerConnectionImpl::sendProtocolError() [0x5c0809e3ac22]
[2025-04-01 18:07:22.398][22][critical][backtrace] [./source/server/backtrace.h:121] #10: Envoy::Http::Http1::ConnectionImpl::dispatchSlice() [0x5c0809e34080]
[2025-04-01 18:07:22.399][22][critical][backtrace] [./source/server/backtrace.h:121] #11: Envoy::Http::Http1::ConnectionImpl::dispatch() [0x5c0809e33562]
[2025-04-01 18:07:22.400][22][critical][backtrace] [./source/server/backtrace.h:121] #12: Envoy::Http::Http1::ServerConnectionImpl::dispatch() [0x5c0809e3a616]
[2025-04-01 18:07:22.402][22][critical][backtrace] [./source/server/backtrace.h:121] #13: Envoy::Http::ConnectionManagerImpl::onData() [0x5c0809e109fc]
[2025-04-01 18:07:22.403][22][critical][backtrace] [./source/server/backtrace.h:121] #14: Envoy::Network::FilterManagerImpl::onContinueReading() [0x5c080a0e5065]
[2025-04-01 18:07:22.404][22][critical][backtrace] [./source/server/backtrace.h:121] #15: Envoy::Network::ConnectionImpl::onReadReady() [0x5c080a07e79e]
[2025-04-01 18:07:22.406][22][critical][backtrace] [./source/server/backtrace.h:121] #16: Envoy::Network::ConnectionImpl::onFileEvent() [0x5c080a07a8cb]
[2025-04-01 18:07:22.407][22][critical][backtrace] [./source/server/backtrace.h:121] #17: std::__1::__function::__func<>::operator()() [0x5c080a084ca6]
[2025-04-01 18:07:22.409][22][critical][backtrace] [./source/server/backtrace.h:121] #18: std::__1::__function::__func<>::operator()() [0x5c080a06dd16]
[2025-04-01 18:07:22.410][22][critical][backtrace] [./source/server/backtrace.h:121] #19: Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb() [0x5c080a06eeb5]
[2025-04-01 18:07:22.411][22][critical][backtrace] [./source/server/backtrace.h:121] #20: event_process_active_single_queue [0x5c080a3818d3]
[2025-04-01 18:07:22.413][22][critical][backtrace] [./source/server/backtrace.h:121] #21: event_base_loop [0x5c080a3804c1]
[2025-04-01 18:07:22.414][22][critical][backtrace] [./source/server/backtrace.h:121] #22: Envoy::Server::WorkerImpl::threadRoutine() [0x5c080982c50e]
[2025-04-01 18:07:22.415][22][critical][backtrace] [./source/server/backtrace.h:121] #23: Envoy::thread::PosixThreadFactory::createPthread()::$_0::__invoke() [0x5c080a3914c4]
[2025-04-01 18:07:22.415][22][critical][backtrace] [./source/server/backtrace.h:123] #24: [0x7411ae6caac3]
Http1::ConnectionImpl 0x32b83fac9c10, dispatching_: 1, dispatching_slice_already_drained_: 0, reset_stream_called_: 0, handling_upgrade_: 0, deferred_end_stream_headers_: 0, processing_trailers_: 0, buffered_body_.length(): 0, header_parsing_state_: Field, current_header_field_: , current_header_value_: 
active_request_: 
, request_url_: /, response_encoder_.local_end_stream_: 0
absl::get<RequestHeaderMapPtr>(headers_or_trailers_): 
current_dispatching_buffer_ front_slice length: 16384 contents: "11111111

nezdolik avatar Apr 10 '25 10:04 nezdolik

Any idea what was the size of the header?

cpakulski avatar Apr 10 '25 19:04 cpakulski

You can see the header in: https://github.com/user-attachments/files/19541163/request.json it's around 65k

bg-ts avatar Apr 11 '25 23:04 bg-ts

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions[bot] avatar May 12 '25 00:05 github-actions[bot]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions[bot] avatar Jun 14 '25 20:06 github-actions[bot]