Some tests in pkg/kates are flakely

[haq204]

Describe the bug Some of the tests in pkg/kates give inconsistent results. So far I've been able to confirm that TestCoherence and TestDeltasWithRemoteDelay are flakely.

To Reproduce

TestCoherence

1. Make sure any containers that kates tests rely are pruned. We're starting from scratch
2. Run Test. You should observe it passing
3. Clear go test cache and run test again. You should no see it fail
4. Run test again. You should now see it pass again

TestDeltasWithRemoteDelay

1. Make sure any containers that kates tests rely are pruned. We're starting from scratch
2. Run Test. You should observe it fail
3. Run Test again. You should observe it passing

Expected behavior Tests should consistently either pass or fail

Additional Context Many tests in the kates package are closer to integration tests in that they rely on external K3s docker container. Most of the inconsistencies in the tests seem to be the result of bad state, tests not properly cleaning up after themselves or certain timing or racing issues when starting up the K3 container.

When running tests, I used the same flags CI does: go test -race -count=1 -timeout 30m -parallel=150