Decide how skipAwait should interact with deletion

[lblackstone]

Problem description

https://github.com/pulumi/pulumi-kubernetes/blob/d7871675b54727f63f1e2254d06653daf06cc8c8/provider/pkg/await/await.go#L463-L511

For resource types that don't have custom await logic, the provider currently ignores the pulumi.com/skipAwait annotation. If this annotation is set, the provider should immediately consider the resource to be deleted.

Suggestions for a fix

Duplicate the logic here: https://github.com/pulumi/pulumi-kubernetes/blob/d7871675b54727f63f1e2254d06653daf06cc8c8/provider/pkg/await/await.go#L446-L447

[blampe]

Edit: tracking separately https://github.com/pulumi/pulumi-kubernetes/issues/3157

[EronWright]

I wonder how replacement would combine with skipAwait on delete and with SSA. Would the Apply call performed by the replacement be racing against finalization of the original?

[blampe]

After a lot of discussion we've decided to punt on this for the time being. The existing behavior is arguably buggy, so we need to be cautious about expanding the behavior to more resources.

To elaborate on why this is buggy, when we delete resources that respects skipAwait (Deployments, Jobs, StatefulSets, Namespaces, Pods, and ReplicationControllers) we don't wait for the deletion to succeed. In cases where a resource is getting delete-before-replaced, this combines with our SSA upsert behavior (https://github.com/pulumi/pulumi-kubernetes/issues/2998) to create a race condition where the resource could still be getting deleted when we upsert.

The annotation currently provides two functions when deleting relevant resources:

1. It makes deletion fast, but this is almost always better handled with a background or orphan delete (https://github.com/pulumi/pulumi-kubernetes/issues/2529).
2. It makes it possible to ignore stuck finalizers, which is generally not recommended but sometimes necessary.

We're currently considering changing the behavior to completely ignore the annotation during deletion. For users who need (1) there's another annotation available. It's unclear how many users actually rely on the annotation for (2) but we assume it's small -- if not, we might need a separate workaround for this case.