Create e2e test for testing at scale

[alexbrand]

We need a suite of tests for testing kismatic at scale. We can leverage the existing framework, but the test itself might need to run on a schedule instead of on every build.

[josephjacks]

What kind of scale testing are we thinking? I know we've tested at ~100 nodes on AWS. Will we try to run at the K8s SLO level (shooting for 5K nodes in 1.5, IIRC)? Will we come close? Would this just run on AWS as that is primarily where our existing e2es run, or can we setup something on Packet as well for bare metal testing?

[alexbrand]

In my mind, the effort here is to automate the ~100 node test so that we can run it on schedule. This will also help in reducing the manual process required for each release, as we won't have to do this kind of tests manually.

We could definitely target Packet if we wanted to. We have the framework for it.