WASM Calling `proxy_send_local_response` twice will stuck remote http client(e.g. curl) forever until timeout or interrupted

[orangetangerine]

Title: WASM Calling proxy_send_local_response twice will stuck remote http client(e.g. curl) forever until timeout or interrupted

Description: I have a wasm running on envoy, and I found http request will be stuck if I call proxy_send_local_response both in on_http_request_headers and on_http_response_headers.

Repro steps: Here is the shortest wasm code:

use log::info;
use proxy_wasm::traits::*;
use proxy_wasm::types::*;

proxy_wasm::main! {{
    proxy_wasm::set_log_level(LogLevel::Debug);
    proxy_wasm::set_root_context(|_| -> Box<dyn RootContext> { Box::new(HttpAuthProxyRoot{})});
}}


struct HttpAuthProxyRoot {}

impl RootContext for HttpAuthProxyRoot {
    fn create_http_context(&self, context_id: u32) -> Option<Box<dyn HttpContext>> {
        Some(Box::new(HttpAuthProxy::new(context_id)))
    }

    fn get_type(&self) -> Option<ContextType> {
        Some(ContextType::HttpContext)
    }
}

impl Context for HttpAuthProxyRoot {}

#[derive(Default)]
struct HttpAuthProxy {
    context_id: u32,
}

impl HttpAuthProxy {
    fn new(context_id: u32) -> HttpAuthProxy {
        HttpAuthProxy { context_id }
    }

    fn test_response(&self) -> Action {
        self.send_http_response(401, vec![], None);
        Action::Continue
    }
}

impl HttpContext for HttpAuthProxy {
    fn on_http_request_headers(&mut self, _num_headers: usize, _end_of_stream: bool) -> Action {
        let ret = self.test_response();
        info!("on_http_request_headers done");
        ret
    }

    fn on_http_response_headers(&mut self, _num_headers: usize, _end_of_stream: bool) -> Action {
        let ret = self.test_response();
        info!("on_http_response_headers done");
        ret
    }

    fn on_log(&mut self) {
        info!("#{} completed", self.context_id);
    }
}

impl Context for HttpAuthProxy {}

envoy docker-compose configuration:

services:
  envoy:
    image: envoyproxy/envoy:v1.24.0
    command: -c /etc/envoy/envoy.yaml -l debug
    hostname: envoy
    ports:
      - "10000:10000"
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml
      - ./target/wasm32-wasi/release:/etc/envoy/proxy-wasm-plugins
    networks:
      - envoymesh
networks:
  envoymesh: {}

and envoy.yaml:

admin:
  address:
    socket_address:
      protocol: TCP
      address: 0.0.0.0
      port_value: 15000
static_resources:
  clusters:
    - name: test_cluster
      type: STATIC
      connect_timeout: 0.250s
      load_assignment:
        cluster_name: test_cluster
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 15000
  listeners:
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 10000
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                codec_type: AUTO
                route_config:
                  name: local_routes
                  virtual_hosts:
                    - name: local_service
                      domains:
                        - "*"
                      routes:
                        - match:
                            prefix: "/"
                          direct_response:
                            status: 200
                            body:
                              inline_string: "Request /hello and be welcomed!\n"
                http_filters:
                  - name: envoy.filters.http.wasm
                    typed_config:
                      "@type": type.googleapis.com/udpa.type.v1.TypedStruct
                      type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
                      value:
                        config:
                          name: "http_headers"
                          vm_config:
                            runtime: "envoy.wasm.runtime.v8"
                            code:
                              local:
                                filename: "/etc/envoy/proxy-wasm-plugins/plugin.wasm"
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

Logs:

And here is log on client side:

I found out that it's caused by this PR https://github.com/envoyproxy/envoy/pull/23049

It occurs since envoy 1.24+ (also Istio 1.16+). If I revert these codes and build, it works again.

I don't know if this behaviour is expected or changed by accident.

It only occurs in my QA cluster (with a higher version of envoy, 1.24+)