envoy icon indicating copy to clipboard operation
envoy copied to clipboard

Envoy closing GRPC stream connection

Open amBunkowski opened this issue 1 year ago • 1 comments

Envoy closing GRPC stream connection after ~15seconds

If i trying with grcpurl I'am getting:

ERROR:
  Code: Internal
  Message: stream terminated by RST_STREAM with error code: NO_ERROR

Config:

admin:
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address: { address: 0.0.0.0, port_value: 8000 }
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                codec_type: AUTO
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: ["*"]
                      typed_per_filter_config:
                        envoy.filters.http.cors:
                          "@type": type.googleapis.com/envoy.extensions.filters.http.cors.v3.CorsPolicy
                          allow_origin_string_match:
                            - safe_regex:
                                regex: \*
                          allow_methods: "GET,POST,PUT,PATCH,DELETE,OPTIONS"
                          allow_headers: "DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,content-type,Range,Authorization,Access-Control-Allow-Origin,keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout"
                          expose_headers: custom-header-1,grpc-status,grpc-message,content-type
                          max_age: "1728000"
                      routes:
                        - match: { prefix: "/" }
                          route: { cluster: sensor_service }
                http_filters:
                  - name: envoy.filters.http.cors
                    typed_config:
                      "@type": ype.googleapis.com/envoy.extensions.filters.http.cors.v3.CorsPolicy
                
                  - name: envoy.grpc_web
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

                access_log:
                  - name: envoy.access_loggers.file
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                      path: /tmp/access.log
                      log_format:
                        text_format_source:
                          inline_string: "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"\n"

  clusters:
    - name: sensor_service
      connect_timeout: 0.25s
      type: LOGICAL_DNS
      dns_lookup_family: V4_ONLY
      lb_policy: ROUND_ROBIN
      typed_extension_protocol_options:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicit_http_config:
            http2_protocol_options:
              # Configure an HTTP/2 keep-alive to detect connection issues and reconnect
              # to the admin server if the connection is no longer responsive.
              connection_keepalive:
                interval: 30s
                timeout: 50s
      load_assignment:
        cluster_name: sensor_service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: host.docker.internal
                      port_value: 9089

Logs: [2024-08-22T15:03:14.900Z] "OPTIONS /sensors.Sensor/TempSensor HTTP/1.1" 200 - "-" 0 0 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "579df928-3b84-4c89-8dd0-e99eac518d12" "localhost:8000" "-" [2024-08-22T15:03:14.901Z] "OPTIONS /sensors.Sensor/HumiditySensor HTTP/1.1" 200 - "-" 0 0 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "dd3dc0cc-ae02-40b8-a5b3-17d2bdc9fc4c" "localhost:8000" "-" [2024-08-22T15:03:14.902Z] "POST /sensors.Sensor/HumiditySensor HTTP/1.1" 200 - "-" 8 148 10044 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "f2e47b59-89e4-43b3-ac6f-6e97f2434ba0" "localhost:8000" "localhost:9089" [2024-08-22T15:03:14.901Z] "POST /sensors.Sensor/TempSensor HTTP/1.1" 200 UT "-" 8 2384 15001 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36" "76a3f36f-fff8-4847-98a7-db48b2aa8f0c" "localhost:8000" "localhost:9089"

amBunkowski avatar Aug 22 '24 15:08 amBunkowski

Sounds like a timeout is firing. You might want to disable the route timeout.

From docs:

timeout (Duration) Specifies the upstream timeout for the route. If not specified, the default is 15s. This spans between the point at which the entire downstream request (i.e. end-of-stream) has been processed and when the upstream response has been completely processed. A value of 0 will disable the route’s timeout.

KBaichoo avatar Aug 23 '24 20:08 KBaichoo

I managed to find solution:

routes: - match: { prefix: "/" } route: timeout: 3600s cluster: sensor_service

Imo documentation is poorly provided with examples. And value 0 in my case did not disable route's timeout

amBunkowski avatar Aug 31 '24 12:08 amBunkowski