sandbox: do retry for wait to remote sandbox controller

[abel-von]

For remote sandbox controllers, the controller process may restart, we have to retry if the error indicates that it is the grpc disconnection.

[k8s-ci-robot]

Hi @abel-von. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Burning1020]

/ok-to-test

[kzys]

/retest

[abel-von]

/cc @mxpv @mikebrow @dmcgowan @fuweid

[abel-von]

/cc @fuweid

[cpuguy83]

I like this, and may argue that it wouldn't hurt to do this in all cases, not just remote sandbox controller.

[cpuguy83]

For reference https://github.com/cpuguy83/containerd-shim-systemd-v1 which does not really make sense as a sandbox controller but would likely be worth it to implement even just for this change.