gloo icon indicating copy to clipboard operation
gloo copied to clipboard

CI Test Flake: [AfterEach] Clustered redis backed rate limiting

Open nfuden opened this issue 3 years ago • 2 comments

Gloo Edge Version

1.13.x (beta)

Kubernetes Version

No response

Describe the bug

Flakes consistently in aftereach

https://00f74ba44b01bed185da38eada35ce74c725f20f82-apidata.googleusercontent.com/download/storage/v1/b/825641009090.cloudbuild-logs.googleusercontent.com/o/log-26eb0502-4374-4330-ad6d-a5f684d7a2c8.txt?jk=AFshE3XnEJnoHcmgsybo-zqHuEwUyD2sU9VFoSFTZ66oNzGXaKazs2c0n6OM_JGN1g7tuELwWCtxGcccVnvZctG4ai2YyUnf2ips33QyVOuRIkTe9XZU5TrxQddAzRWK_nJ7KvsAWbzwGRKODq-IWkFWrZP6YexeYeMghvJ8owzMkadX40gzyVTR9ntzitIitg_eWPu6pgzNxKJrLon19tAWRHxfQjFkUFUIo6zbd4C4BKpXeoMTjFF0Yy0JqnK_CIarrNRp3nMds-v1cjocqwn_IAQqoPo_uwaZ5ADuEoonOuhwOG1cMd-CR7ePOthDqDGM-ZR9Ux6DdFoPo7tEdQTBXzX2oa1iolbeSbmP4_C02hlAgkTg9ieqHoAvfK0b2SQQ35Fdrz5rmI-H3eFFQ70GsyhzMqf711Cry8M99FrnBAWLpd_fFXxRt1wxK9Ki80xgn-gTBCS1486eHKYZkcX7lT9EyKl0Pba--Q0jUHqFWXQ8sk4K8au1V5J5MejJzuOcbqm6VdGs62q_H0ywCRDdjD5ithaVfGblXjCOYawgwBJG0NT0pcKL3k_u6dmC6a2PvbFXTViZ-cFThRe3Pi1A3chknP-yOeA_52jAprKO4Qa0dQN-EbLZpAnU-gtEC27ah395IqhpaYMv7oqKZtaDRW7dizzUWtnxFGx8NvZS1GAgSU3IV7kzMXrq3moDEn-wwvE_bE_YQPqpF-UVjpg1xmLHo8XE1HKBdWhAgP8KQS4vWXsqK2G_IHktY4cGC9MXGazmzyPLrEwa0T4CP_as6mcFDkMzqM8x8AVsnQc54aLSRhf9huaBzyr6BAaf1HOw9fgmx2K83ShsmSE4iuu9DFgHOKLG1_5YdJ9oPQXsosczpf35yOMNLcl1aoLrhiY5Pj50TacmCC0nHCEnKoLhYXccx-n9FX8_mgz6i3QdZmFjWyr3Ca00BBt-scOw7mnxrYAa1pHUZVO0yKgeE2ONeX6gnRJ2woxho8EG8mHNVkLZdknr_8Ol9ittYvCUyWKVDo5kk5BJN53pM2Peb-LhspxP4nLkCdsGgwnVCCZ53ISlqsppRtQRKyNL1Do94gkm&isca=1

Steps to reproduce the bug

flakes

Expected Behavior

doesnt flake

Additional Context

No response

nfuden avatar Sep 16 '22 19:09 nfuden

Step #12 - "test": [2022-09-16 18:57:29.569][86309][critical][assert] [external/envoy/source/extensions/filters/common/ratelimit/ratelimit_impl.cc:77] assert failure: response->overall_code() != envoy::service::ratelimit::v3::RateLimitResponse::UNKNOWN.

This line is the culprit. @Bslabe123 and I thought this was fixed by changing the rate-limiter to return nil instead of envoy::service::ratelimit::v3::RateLimitResponse::UNKNOWN in this recover

jbohanon avatar Sep 16 '22 19:09 jbohanon

The fix was released in 0.7.4 of the rate-limiter, and 1.12 (where this flake occurred) is on 0.7.3

jbohanon avatar Sep 16 '22 19:09 jbohanon

@inFocus7 I believe your recent PRs in solo-projects resolved this flake. Can you confirm and if so, close the issue?

sam-heilbron avatar Oct 02 '22 19:10 sam-heilbron