envoy icon indicating copy to clipboard operation
envoy copied to clipboard

`@envoy//test/common/memory:memory_release_test ` is very flakely TIMEOUTing

Open phlax opened this issue 1 year ago • 11 comments

Quite a few PRs are failing compile-time-options with a TIMEOUT on posted test

postsubmit seems to be mostly passing but its hitting a lot of PRs

i think the change that created the issue is here https://github.com/envoyproxy/envoy/pull/30353

https://dev.azure.com/cncf/envoy/_build/results?buildId=162619&view=logs&j=e969334a-0e55-5c18-ac96-8b546753391e&t=4122df52-7ea9-5745-8c7d-f6b6f2c73149&l=134

phlax avatar Feb 16 '24 19:02 phlax

cc @nezdolik

phlax avatar Feb 16 '24 19:02 phlax

cc @ohadvano

phlax avatar Feb 16 '24 19:02 phlax

@phlax i will create revert patch and open a test patch (later) to try out various parameters of memory allocation size with CI environment. cc @KBaichoo @yanavlasov

nezdolik avatar Feb 16 '24 23:02 nezdolik

Might be related to compile time options (looking at a sample of failed test):

  • https://dev.azure.com/cncf/envoy/_build/results?buildId=162620&view=results
  • https://dev.azure.com/cncf/envoy/_build/results?buildId=162609&view=logs&j=e08c1e20-d729-552d-2215-5529dbdb5756
  • https://dev.azure.com/cncf/envoy/_build/results?buildId=162619&view=logs&s=4f7d954b-a765-565f-91a2-c04870dab43f&j=e969334a-0e55-5c18-ac96-8b546753391e
  • https://dev.azure.com/cncf/envoy/_build/results?buildId=162029&view=logs&j=e969334a-0e55-5c18-ac96-8b546753391e&t=4122df52-7ea9-5745-8c7d-f6b6f2c73149

I think the system doesn't work as expected with gperftools/tcmalloc; it might make sense to just enable it for tcmalloc/tcmalloc so we can get more deployment experience with this and figure out the gperftools piece as a follow up or a todo.

KBaichoo avatar Feb 20 '24 15:02 KBaichoo

@nezdolik , will you fix https://github.com/envoyproxy/envoy/pull/30353 for merging into envoyproxy:main ?

anvo1115 avatar Mar 12 '24 13:03 anvo1115

@anvo1115 am working on fixing it in https://github.com/envoyproxy/envoy/pull/32589, for now memory releasing will not be supported for gperf tcmalloc. I work on the fix on my free time (is not part of my daily job).

nezdolik avatar Mar 12 '24 13:03 nezdolik

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions[bot] avatar Apr 11 '24 16:04 github-actions[bot]

not stale. we are looking forward for a fix

nichd147 avatar Apr 17 '24 07:04 nichd147

im wondering if there is a better issue to track - this one was for the ci issue which was resolved by reverting

phlax avatar Apr 17 '24 07:04 phlax

right, sorry. missed pr

nichd147 avatar Apr 17 '24 08:04 nichd147

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions[bot] avatar May 17 '24 12:05 github-actions[bot]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

github-actions[bot] avatar May 24 '24 12:05 github-actions[bot]