kibana
kibana copied to clipboard
elastic
Flaky alert assignment tests
Summary
Fix flaky alert assignments tests. I split assignments tests into two groups: tests with one assignee available and tests with multiple assignees.
Right now there is a flakiness in tests with multiple assignees. Most probably it is happening because we do multiple login calls in a row to make sure we activate different users to make them available for assignments:
// Login into accounts so that they got activated and visible in user profiles list
login(ROLES.t1_analyst);
login(ROLES.t2_analyst);
login(ROLES.t3_analyst);
login(ROLES.soc_manager);
login(ROLES.detections_admin);
login(ROLES.platform_engineer);
These tests are tend to be flaky and it is possible that kibana operations team will skip those. To make sure that we run basic cypress verification of alert assignments feature we decided to add tests with only one assignee available (current user) which allows us to avoid multiple consecutive login calls.
Also, as part of these changes I removed unnecessary logins and un-skipped https://github.com/elastic/kibana/issues/176529
Checklist
Delete any items that are not applicable to this PR.
- [x] Flaky Test Runner was used on any tests changed
Thank you for the review @rylnd!!
I think the only needed change here is removal of the redundant single-user tests; everything else is a step in the right direction.
I agree with redundancy of update assignee in single user case and will update/remove unnecessary test cases.
I was looking for examples of where these tests failed, to validate the multi-user hypothesis and see if there wasn't some information in the error/failure. However, the original skip PR linked here only shows that ES threw a 503 during the test, and the flaky test runner seemingly only had timeouts and not any legitimate failures.
If there are particular error messages that we're basing this PR on, it would be great to call those out both in this PR and the "skipped test" issue, for posterity.
Yes, the 503 error is what casing the issue. It happens within beforeEach block on deletion of indices and lists. We do exactly the same steps as in all other tests except in our case we do multiple login calls to activate multiple accounts. That's why this is the only reason I can think of that could cause that internal error. While I will be investigating that issue further, I would like to have at least some stable tests covering assignments functionality.
@MadameSheema we discussed the idea of testing one user vs multiple here, and the potential loss of coverage, and it was argued that having:
- a cypress test to verify that a single user assignment propagates correctly to the UI
- a cypress test to verify that multiple user assignments propagate correctly to the UI
- multiple jest tests to test component behavior for the myriad users/roles assigned
would provide the same coverage as all the previous cypress tests, which much less cost/downside. Do you agree with that?
:yellow_heart: Build succeeded, but was flaky
- Buildkite Build
- Commit: 757591d6ae45724fc7f3eacbd2dda06a66ea5430
Failed CI Steps
Test Failures
- [job] [logs] Defend Workflows Cypress Tests on Serverless #10 / User Roles for Security Complete PLI with Endpoint Complete addon for role: endpoint_operations_analyst should have access to response action: processes should have access to response action: processes
Metrics [docs]
✅ unchanged
History
- :yellow_heart: Build #195940 was flaky 161649f77acb83e6b36cd954e8dc05452cbdea86
- :yellow_heart: Build #195846 was flaky 915ea6f57196932f496a013d73d2622c00202754
- :yellow_heart: Build #195319 was flaky c098902c2515f4b4877bd0b0841f89e6c711b50d
- :broken_heart: Build #195269 failed 093e8bb0dcdd2aa873339799479b9cf8107d6634
- :green_heart: Build #194100 succeeded 0f6d112c5bbe68657c108fd94f469dee6133b525
- :green_heart: Build #194013 succeeded 432f4de0d5aa7a868d30aca87a86706a044bcd07
To update your PR or re-run it, just comment with:
@elasticmachine merge upstream
cc @e40pud
