[long-running] [collecting info] additional rules in default config

[cppforlife]

this issue is to track proposed additional deploy config that shuold be added to kapp. current default config can be seen via kapp deploy-config.

[cppforlife]

BogdanL reported unexpected diffing of PVs: https://kubernetes.slack.com/archives/CH8KCCKA5/p1573590861170100. Suggestion was to add PV claimRef and some other fields as rebase rules (https://gist.github.com/cppforlife/149872f132d6afdc6f0240d70f598a16)

[cppforlife]

• [x] suggestion to add Job as a target for templatRules for ConfigMap/Secret
• [x] fallback on existing spec.healthCheckNodePort for v1/Service

[cppforlife]

template rules for CronJob (https://gist.github.com/cppforlife/56d44562d6b44e529a3c5299bb2d8943)

[cppforlife]

aggregated rbac rules: https://github.com/k14s/kapp/issues/140#issuecomment-684104068

[caphrim007]

not sure if this is the appropriate location to put this, but HorizontalPodAutoscaler will modify the HPA annotations like so

  0,  0   apiVersion: autoscaling/v1
  1,  1   kind: HorizontalPodAutoscaler
  2,  2   metadata:
  3     -   annotations:
  4     -     autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True",...count"}]'

and kapp will always remove these and HPA will just add them back. I (personally) include a rebase rule to tell kapp to ignore this (because the HPA manages it). Here's the rule I use

      - path: [metadata, annotations, autoscaling.alpha.kubernetes.io/conditions]
        type: copy
        sources: [existing, new]
        resourceMatchers:
          - apiVersionKindMatcher: {apiVersion: autoscaling/v1, kind: HorizontalPodAutoscaler}

[gabegorelick]

As discussed in https://kubernetes.slack.com/archives/CH8KCCKA5/p1611785343055300, ValidatingWebhookConfiguration and MutatingWebhookConfiguration need to be applied after the Service they reference.

This is breaking some deploys of Gatekeeper (ones that use https://github.com/open-policy-agent/gatekeeper/blob/master/deploy/experimental/gatekeeper-mutation.yaml, separate from the issues in https://github.com/vmware-tanzu/carvel-kapp/issues/52) since kapp seems to block on applying the ValidatingWebhookConfiguration before it attempts to apply the Service (not sure why, it would maybe be fine if they were done in parallel). This leads to the following error:

Retryable error: Creating resource namespace/gatekeeper-system (v1) cluster: Internal error occurred: failed calling webhook "check-ignore-label.gatekeeper.sh": Post "https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admitlabel?timeout=3s": service "gatekeeper-webhook-service" not found (reason: InternalError)

[100mik]

Looks like something that might belong here #514