fix: retry on errors when watching pods

[mikedld]

Fixes: #8658

Description If timeout (or some network error) occurs while waiting for a pod initialization or termination event, e.g. when build takes a long time, skaffold becomes stuck and never finishes the operation. Use retry watcher to handle the errors gracefully.

This PR is based on the patch I posted in #8658 last year; never got any feedback on it there so decided to go ahead. I'm using this patch since then and it works fine on my end. To reiterate,

Also note that the same issue affects WaitForDeploymentToStabilize (and probably some other places where Watch is used) but I can't test it so I didn't patch it.

I only managed to fix exising unit test, not add any new test(s), as I'm not at all comfortable with Go. If that's an issue, I'm okay with someone else picking this up.

[google-cla[bot]]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[certifiedloud]

How can we encourage this fix to be merged? This issue is causing significant issues for skaffold users who want to utilize kaniko.

[alphanota]

@mikedld Thank you for this PR. Would you mind fixing the conflicting files and that the PR is synced to skaffold main?