make resource annotations more resilient to being clobbered by external actors

[cppforlife]

per investigation with @jorgemoralespou, tekton copies over all annotations from Pipeline to PipelineRun resource. in our case both Pipeline and PipelineRun resources are being directly created by the user via kapp (kapp annotates both with various resource specific metadata eg kapp.k14s.io/identity). when tekton controller replaces metadata annotations kapp no longer believes that this resource is being directly managed by the user (it still tracks it but as a cluster created resource).

one solution that may solve this problem is to use "unique" annotations (eg kapp.k14s.io/identity.<guid>) so that if some controller does copy over annotations from other resources, one can potentially still detect "true identity" of the resource. this of course does not cover the case when controllers are deleting annotations (instead of merging).

another possibility potentially is to keep metadata outside of the resource but that comes at a price of doing your own garbage collections, figuring sizing constraints etc.

[cari-lynn]

Adding more detail as to the problem:

Some labels kapp uses to track resources are:

• kapp.k14s.io/app to track an app using an id
• kapp.k14s.io/identity to store info of the resource (combination of namespace, kind, etc), that can be used to compare against the resource returned from the cluster to see if it matches. It is used to determine transientiveness (if a app is owned by kapp or by the cluster) since resources labels are sometimes copied.

In tektons case, a simplifed manifest deployed as:

kind: Pipeline
metadata:
  labels:
    kapp/app: "pipeline"
    kapp/identity: "pipeline"
---
kind: PipelineRun
metadata:
  labels:
    kapp/app: "pipeline-run"
    kapp/identity: "pipeline-run"

after its deployed, will be changed to:

kind: Pipeline
metadata:
  labels:
    kapp/app: "pipeline"
    kapp/identity: "pipeline"
---
kind: PipelineRun
annotations:
  labels:
    kapp/app: "pipeline"
    kapp/identity: "pipeline" <-- the 'identity' was overwritten

The PipelineRun annotations are overwritten and lost.

The problem shows up when kapp tries to delete the resource. kapp queries the cluster for this PipelineRun resource it checks the kapp/identity label to verify it matches the resource type, namespace, name etc it expects, and since it doesn't, kapp thinks its a cluster owned resource. kapp then hangs because it is waiting for the cluster to delete it.

A workaround is possible, if you include the --apply-ignored flag to force the resource delete. However, it is not a solution since it may apply more changes than just the one delete that is necessary.

[cari-lynn]

From the solution @cppforlife mentioned above regarding adding kapp.k14s.io/identity.<guid> label to all resources. Then the result of copying labels won't overwrite the identity label, instead it would add a label. Then kapp could check whether a resource has at least one identity label that 'matches' the resource to know its a kapp owned resource.

Please add other ideas to this thread :)

[renuy]

AI: Try and reproduce this and understand the issue more in depth. to reproduce: deploy two resoures one with kapp, another with kubectl. and copy the identity to the second resource.