gc-tag should use label, not annotation

[anguslees]

Garbage collection works by listing everything with the gc-tag. In a busy cluster, we really want that filter to happen server-side and ideally using an index of some sort.

That means we should use a Kubernetes label, not an annotation.

I think this will require a two-step migration plan (write both but continue to read annotation; release; drop support for annotation; release).

[anguslees]

@gotwarlost points out in qbec's version of garbage collection that the Service->Endpoints controller copies labels across, but not annotations.

We will need to ensure that we don't garbage collect Endpoints unnecessarily (deserves an explicit integration test case). The fix probably involves recording "kubecfg created this object as <group/kind>" annotation, and then double-checking the resource is both marked and the original kind before deleting. Note this also means Deployments will only be deleted via the original apigroup (currently kubecfg's uid logic makes this safe).

[gotwarlost]

nice trick with the annotation check. I'll incorporate some version of this in qbec soon. Arguably I already have this info in the last-applied annotation that stores the pristine object.

[kgtw]

Hi @anguslees

Any update on the progress of phase 2 for this, having kubecfg retrieve every resource for each resource type is extremely inefficient when the cluster has even a few hundred unique resources.