End-to-end test suite refactor

[snay2]

Problem

Our current integration test suite is written in Bash and is becoming difficult to maintain. We would like to be able to support more diverse cluster types and NTH installations, but adding such support is cumbersome and requires a lot of duplicated work in each test. We also want to make it easier to create new tests as we add new features.

Proposal

Create a integration test framework, written in Go, that includes cluster-agnostic drivers, common functions (e.g., "simulate an ITN", "verify that the pod has been evicted"), and reusable components that are easy to combine into relevant end-to-end test cases. Then, rewrite our existing tests using this framework and incorporate them in our CI/CD system. Once we are confident that the new test suite is performing at least as well as the existing test suite, remove the old tests from the repo.

Tenets

• Cluster agnostic: User can choose any of the following:
  • Fully-managed local kind cluster (setup + teardown, setup, reuse existing)
  • Partially-managed EKS cluster (setup, reuse existing)
• Extensible: Creating a new test does not require copying any boilerplate code. Tests should be designed with reusable components.
• Consumable test output: Test output is accessible to human and automated test runners
• Fast: Tests run quickly to provide immediate, actionable feedback to developers

NTH v2

Work is underway for version 2 of NTH (read the design proposal here). This new test suite should support v2, ideally in a way that even if the framework needs to be updated, the individual tests can remain the same.

[jillmon]

As of Jan 19, 2023, this is no longer a work in progress. We are still considering this as a potential future enhancement, but have no concrete plans or set dates at this time.