Tests need support for simulated errors

[theunrepentantgeek]

We should cover various corner cases being able to inject certain failures. Some of this should be achievable using our existing envtest + recording framework and some may need some additional framework updates.

Ideas:

• [ ] DeploymentAlreadyExists when we attempt to create it?
• [ ] InternalServerError for a period of time from a service.
• [ ] "Create" a resource that the operator doesn't know about but which already exists.
• [ ] Deployment fails after some polling due to 5xx error.
• [ ] Throttling from a service.
• [ ] Simulated errors saving changes to spec or status (kube-apiserver instability)
• [ ] Simulated loss of entire status

Originally logged as #1726 by @matthchr

[matthchr]

This is still true and something we should investigate further

[matthchr]

No status change from above, although a lot of this can be covered with our existing unit tests so the need isn't urgent.

[theunrepentantgeek]

Still something we want to do.

[matthchr]

Still something we want to do, I have some ideas about how we could do this related to the "manually pump the reconciler" tests that I saw while helping with Karpenter.

[matthchr]

No change from above

[theunrepentantgeek]

Would be nice to have.

[matthchr]

We still want to do this. I've been somewhat inspired by some stuff that Karpenter does but haven't had the time to port it over for us.

[matthchr]

No change from above.