one icon indicating copy to clipboard operation
one copied to clipboard

`Development` - Make cypress tests more robust

Open vichansson opened this issue 2 years ago • 0 comments

Description A lot of the tests that fail in our cypress readiness suite are so called "flaky" that is, fail due to some unpredictable condition E.g. random network error, lack of resources, temporary outage etc... these are of course things we can't directly predict or so easily prevent. Instead we should adapt our testing environment to better deal with these events by implementing things such as Cypress Retry-ability. This would provide us with a lot more consistency in our testing reports by eliminating false positives.

Overall changes & Implementation strategy

  1. Make tests more independent
    • Each test should be self-contained and not rely on external factors or other tests to run successfully. This will help prevent cascading failures and make our tests more robust.
  2. Track and manage resources
    • Identify all the resources required for each test, such as virtual machines, hosts, and other dependencies. Create these resources automatically when the test starts, and clean them up afterwards. This will ensure that tests don't interfere with one another and leave behind unwanted stuff.
  3. Implement retry-ability
    • Our tests should be designed to handle unexpected failures gracefully. We can achieve this by introducing retry mechanisms that allow tests to recover from transient errors and network glitches etc... By retrying failed tests, we can generate more accurate and consistent results.

Progress Status

  • [ ] Code committed
    • [x] Cypress cleanup utility https://github.com/OpenNebula/development/pull/2285
    • [ ] Rewrite tests to allocate all necessary resources on start
    • [ ] Rewrite tests to use new cleanup functionality
    • [ ] Implement cypress retry-ability
      • [ ] Add video configuration for failing tests
  • Issues found in test suites
    • [ ] Images tests are testing DEV_PREFIX attribute without creating the image using the GUI
    • [ ] Adds graph/chart tests
      • related to https://github.com/OpenNebula/one/issues/6788
    • [ ] Test number of services instantiated
      • related to https://github.com/OpenNebula/one/issues/6716
  • [ ] Testing - QA
  • [ ] Documentation (Release notes - resolved issues, compatibility, known issues)

vichansson avatar Jul 24 '23 17:07 vichansson