troubleshoot icon indicating copy to clipboard operation
troubleshoot copied to clipboard

Increase e2e test coverage to allow better testing of PRs

Open banjoh opened this issue 2 years ago • 0 comments

Describe the rationale for the suggested feature.

PRs raised, especially by dependabot, will need to be tested better to increase confidence before approval & merging.

Describe the feature

  • Add more collectors & analysers in both support-bundle and preflight e2e specs
  • Run different versions of k8s in a matrix to allow executing troubleshoot applications in different k8s versions
  • Run tests against different OS distros and version (kernels) that we support so that we catch regressions such as this one
  • Run e2e tests against different k8s distros. Openshift is for example quite restrictive and it would be good to cover some basic use cases.
  • Launch, where possible, ancillary systems such as databases, a web server, storage providers (ceph, longhorn) etc. They can be run within the test k3s cluster or as docker-compose containers depending on which is the most appropriate approach. Host collectors will require special attention.
  • Rethink how we assert test results. Using jq might be sufficient, but is there a better way?
  • Add tests that target RBAC scenarios e.g one that would catch https://github.com/replicatedhq/troubleshoot/issues/1079 bug.

Describe alternatives you've considered

Leave things as they are and trust the review process solely.

Additional context

There is higher confidence of the correctness of PRs authored by contributors, but dependabot PRs may introduce bugs that are hard to spot.

banjoh avatar Dec 06 '22 11:12 banjoh