api-services-portal icon indicating copy to clipboard operation
api-services-portal copied to clipboard

Fix flaky rate limiting test

Open github-actions[bot] opened this issue 1 year ago • 3 comments

Stats: { "suites": 88, "tests": 732, "passes": 731, "pending": 0, "failures": 1, "start": "2024-04-12T22:46:54.931Z", "end": "2024-04-12T23:49:47.458Z", "duration": 2397363, "testsRegistered": 732, "passPercent": 99.86338797814207, "pendingPercent": 0, "other": 0, "hasOther": false, "skipped": 0, "hasSkipped": false }

Failed Tests: "verify rate limit error when the API calls beyond the limit"

Run Link: https://github.com/bcgov/api-services-portal/actions/runs/8668759301

github-actions[bot] avatar Apr 12 '24 23:04 github-actions[bot]

These tests continue to fail occasionally (e.g. #1174 / https://github.com/bcgov/api-services-portal/actions/runs/10802153362), with a 429 Too Many Requests being returned when a 200 is expected.

There are a 6 tests with the same it('verify rate limit error when the API calls beyond the limit') so hard to track without looking at each run output (and they are not preserved very long), but in recent runs I have seen failure on the following, so perhaps they are all a bit flaky:

  • describe('Manage Control-Rate Limiting Spec for Service as Scope and Redis Policy')
  • describe('Manage Control-Apply Rate limiting to Global and Consumer at Service level')
  • describe('Manage Control-Apply Rate limiting to Global and Consumer at Route level')

Might just need a little longer wait() to ensure the policy has time to be applied.

rustyjux avatar Sep 10 '24 23:09 rustyjux

Aidan suggests changing the test (or the rate limit update methods) to look for a successful toast to be displayed. This could be a reusable command which gets a toast and checks for the message content. Should be able to set a long timeout for the get toast, since there should be an error/fail toast in case of error.

rustyjux avatar Sep 11 '24 16:09 rustyjux

Priority: Existing commitments and bugs

However, I believe rate limiting works just fine and this is simply a matter of improving the reliability of our tests. So not the highest priority.

rustyjux avatar Oct 08 '24 16:10 rustyjux