E2e http_tunneling_test flake

[jbohanon]

Gloo Edge Version

1.15.x (beta)

Describe the bug

Step #8 - "run-e2e-tests":   [FAILED] in [AfterEach] - /workspace/gloo/test/e2e/http_tunneling_test.go:476 @ 07/28/23 17:34:04.421
Step #8 - "run-e2e-tests":   << Timeline
Step #8 - "run-e2e-tests": 
Step #8 - "run-e2e-tests":   [FAILED] Unexpected error:
Step #8 - "run-e2e-tests":       <*errors.errorString | 0x72bb630>: {s: "EOF"}
Step #8 - "run-e2e-tests":       EOF
Step #8 - "run-e2e-tests":   occurred
Step #8 - "run-e2e-tests":   In [AfterEach] at: /workspace/gloo/test/e2e/http_tunneling_test.go:476 @ 07/28/23 17:34:04.421
Step #8 - "run-e2e-tests": 
Step #8 - "run-e2e-tests":   Full Stack Trace
Step #8 - "run-e2e-tests":     github.com/solo-io/gloo/test/e2e_test.connectProxy.func2()
Step #8 - "run-e2e-tests":     	/workspace/gloo/test/e2e/http_tunneling_test.go:476 +0x1e3
Step #8 - "run-e2e-tests":     created by github.com/solo-io/gloo/test/e2e_test.connectProxy
Step #8 - "run-e2e-tests":     	/workspace/gloo/test/e2e/http_tunneling_test.go:470 +0xe06

[jbohanon]

https://github.com/solo-io/gloo/pull/8487

[sam-heilbron]

Logs: https://storage.googleapis.com/solo-public-build-logs/logs.html?buildid=1e9d12a3-d955-4b3d-9cae-924b85dbdeed Error:

 < HTTP/1.1 200 OK
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   < content-type: text/plain; charset=UTF-8
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   < cache-control: no-cache, max-age=0
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   < x-content-type-options: nosniff
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   < date: Sun, 14 Apr 2024 23:35:14 GMT
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   < server: envoy
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   < transfer-encoding: chunked
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   < 
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   { [13 bytes data]
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   
100     3    0     3    0     0    267      0 --:--:-- --:--:-- --:--:--   272
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   * Connection #0 to host 127.0.0.1 left intact
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   OK
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   [2024-04-14 23:35:14.933][25571][warning][config] [external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:155] StreamAggregatedResources gRPC config stream to xds_cluster closed: 13, 
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   [2024-04-14 23:35:14.933][25571][info][main] [external/envoy/source/server/server.cc:1011] exiting
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   [38;5;9m[FAILED][0m in [AfterEach] - /workspace/gloo/test/e2e/http_tunneling_test.go:463 [38;5;243m@ 04/14/24 23:35:14.936[0m
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   [38;5;243m<< Timeline[0m
Step #1 - "run-tests": Step #8 - "run-e2e-tests": 
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   [38;5;9m[FAILED] error writing from upstream to envoy[0m
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   [38;5;9mIn [1m[AfterEach][0m[38;5;9m at: [1m/workspace/gloo/test/e2e/http_tunneling_test.go:463[0m [38;5;243m@ 04/14/24 23:35:14.936[0m
Step #1 - "run-tests": Step #8 - "run-e2e-tests": 
Step #1 - "run-tests": Step #8 - "run-e2e-tests":   [38;5;9mFull Stack Trace[0m
Step #1 - "run-tests": Step #8 - "run-e2e-tests":     github.com/solo-io/gloo/test/e2e_test.connectProxy.func1()
Step #1 - "run-tests": Step #8 - "run-e2e-tests":     	/workspace/gloo/test/e2e/http_tunneling_test.go:463 +0x325
Step #1 - "run-tests": Step #8 - "run-e2e-tests":     created by github.com/solo-io/gloo/test/e2e_test.connectProxy in goroutine 4669
Step #1 - "run-tests": Step #8 - "run-e2e-tests":     	/workspace/gloo/test/e2e/http_tunneling_test.go:450 +0xe85