meta-protocol-proxy icon indicating copy to clipboard operation
meta-protocol-proxy copied to clipboard

Published 20 hours ago • verified publisher icon aeraki-mesh

压测aeraki.meta_protocol.filters.istio_stats会导致envoy发生crash

Open wuyouxia123 opened this issue 2 years ago • 9 comments

测试步骤：

  1. 测试demo image
  2. jemter 并发数测试为500进行压测 image
  3. 服务端istio-proxy会出现crash
  4. 删除envoyfilter中的aeraki.meta_protocol.filters.istio_stats配置，重新压测，压测通过 image

出错log： 2023-09-21T07:56:39.443577Z critical envoy assert external/envoy/source/common/stats/symbol_table.cc:497 assert failure: search != decode_map_.end(). Details: no such symbol thread=30 2023-09-21T07:56:39.443887Z critical envoy backtrace external/envoy/source/server/backtrace.h:104 Caught Aborted, suspect faulting address 0x53900000014 thread=30 2023-09-21T07:56:39.443935Z critical envoy backtrace external/envoy/source/server/backtrace.h:91 Backtrace (use tools/stack_decode.py to get line numbers): thread=30 2023-09-21T07:56:39.443949Z critical envoy backtrace external/envoy/source/server/backtrace.h:92 Envoy version: 8d280ad75d9b417f734ef0e261e29f230260576b/1.26.4/Modified/DEBUG/BoringSSL thread=30 2023-09-21T07:56:39.739249Z critical envoy assert external/envoy/source/common/stats/symbol_table.cc:688 assert failure: mem_block.capacityRemaining() == 0. thread=32 image

wuyouxia123 avatar Sep 21 '23 08:09 wuyouxia123

测试环境中 accessLogFormat 配置贴出来看一下呢

zhaohuabing avatar Sep 21 '23 08:09 zhaohuabing

测试环境中 accessLogFormat 配置贴出来看一下呢

没有配置accesslog，使用istio默认的accesslog

wuyouxia123 avatar Sep 21 '23 08:09 wuyouxia123

使用这个试试

https://github.com/aeraki-mesh/aeraki/blob/25c89759986ef7858af011b1c3ead87b6e5f0571/test/e2e/common/istio-config.yaml#L17

accessLogFormat: "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)%
     %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\"
     %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER%
     %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"

zhaohuabing avatar Sep 21 '23 08:09 zhaohuabing

使用这个试试

https://github.com/aeraki-mesh/aeraki/blob/25c89759986ef7858af011b1c3ead87b6e5f0571/test/e2e/common/istio-config.yaml#L17

accessLogFormat: "[%START_TIME%] %REQ(X-META-PROTOCOL-APPLICATION-PROTOCOL)%
     %RESPONSE_CODE% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\"
     %BYTES_RECEIVED% %BYTES_SENT% %DURATION% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(X-REQUEST-ID)%\" %UPSTREAM_CLUSTER%
     %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %ROUTE_NAME%\n"

配置accesslog后还是有同样的问题和内存不足的问题 image image

wuyouxia123 avatar Sep 21 '23 09:09 wuyouxia123

Envoy stats 是比较耗内存的，你把内存调大试一下。

zhaohuabing avatar Sep 22 '23 06:09 zhaohuabing

提高内存仍然有同样的问题，且压测时未看到内存有明显提升，且未超过分配值

  1. 内存分配 image

  2. 调整测试demo为并发2次，持续300s，会压测一段时间后发生crash image

  3. envoy占用内存未见明显提升 客户端envoy pid：3348825 服务端envoy pid：3346735 image image

wuyouxia123 avatar Sep 25 '23 12:09 wuyouxia123

@wuyouxia123 请问这个问题解决了吗？

zhaohuabing avatar Oct 10 '23 04:10 zhaohuabing

没有

wuyouxia123 avatar Oct 10 '23 05:10 wuyouxia123

同样出现了这个crash以及OOM在1.4.2 version 用wrk2压测meta_thrift demo中，请问有没有办法解决呢？

yizhengx avatar Apr 11 '24 02:04 yizhengx