Land initial groundwork for better self-hosted testing infrastructure

[hubertdeng123]

We should have a good idea of where we want testing for self-hosted to be in our pipeline, and outline a process for devs to write their own tests for self-hosted. The first step here would be to rewrite our entire testing suite in a framework that is more friendly. If we ourselves struggle to write tests here, how can we expect others to willingly shoulder this burden with us?

### Tasks
- [ ] rewrite current e2e tests to a more friendly testing framework
- [ ] measure flakiness of self-hosted e2e tests and ensure it is under <10%
- [ ] lay out new groundwork to re-add testing in the deployment pipeline