Automatically retry when there are unexpected k8s errors or other infrastructure-y errors

[augray]

When k8s errors happen (like is detected here), we should ideally retry automatically (i.e. even if retry settings were not explicitly added) as these are likely to be transient errors. We might want to make it possible to configure the func such that we don't automatically retry such things though--maybe certain workloads are doing some things it would be dangerous to do twice (make_ten_thousand_dollar_purchase() 😆 ).