howto debugging reconciling loop

[eumel8]

What happened?

Using crossplane provider-helm to deploy vcluster chart. The installation goes never done due non ready state:

NAME        CHART      VERSION   SYNCED   READY   STATE      REVISION   DESCRIPTION        AGE
vc1-8d6xs   vcluster   0.15.7    True     False   deployed   763        Upgrade complete   16m

The debug log shows not exactly what is changed or missing:

2023-09-12T19:28:10.548Z	DEBUG	provider-helm	Looks like there are no changes for [ClusterRoleBinding vc-vc1-vcluster-v-vc1] %!q(MISSING)
2023-09-12T19:28:10.559Z	DEBUG	provider-helm	Looks like there are no changes for [Role vc1-vcluster] %!q(MISSING)
2023-09-12T19:28:10.574Z	DEBUG	provider-helm	Looks like there are no changes for [RoleBinding vc1-vcluster] %!q(MISSING)
2023-09-12T19:28:10.584Z	DEBUG	provider-helm	Looks like there are no changes for [Service vc1-vcluster] %!q(MISSING)
2023-09-12T19:28:10.594Z	DEBUG	provider-helm	Looks like there are no changes for [Service vc1-vcluster-headless] %!q(MISSING)
2023-09-12T19:28:10.606Z	DEBUG	provider-helm	Patch [StatefulSet vc1-vcluster vc1] %!q(MISSING) in namespace %!s(MISSING)
2023-09-12T19:28:10.671Z	DEBUG	provider-helm	updating status for upgraded release for [vc1-vcluster]
2023-09-12T19:28:10.711Z	DEBUG	provider-helm	Successfully requested update of external resource	{"controller": "managed/release.helm.crossplane.io", "request": "/vc1-8d6xs", "uid": "e7e9f382-6cb7-4e3c-bf5c-a5e28793bc76", "version": "72974825", "external-name": "vc1-vcluster", "requeue-after": "2023-09-12T19:38:10.711Z"}
2023-09-12T19:28:10.712Z	DEBUG	events	Successfully requested update of external resource	{"type": "Normal", "object": {"kind":"Release","name":"vc1-8d6xs","uid":"e7e9f382-6cb7-4e3c-bf5c-a5e28793bc76","apiVersion":"helm.crossplane.io/v1beta1","resourceVersion":"72974825"}, "reason": "UpdatedExternalResource"}
2023-09-12T19:28:10.723Z	DEBUG	provider-helm	Reconciling	{"controller": "managed/release.helm.crossplane.io", "request": "/vc1-8d6xs"}
2023-09-12T19:28:10.723Z	DEBUG	provider-helm	Connecting	{"request": "vc1-8d6xs"}
2023-09-12T19:28:10.739Z	DEBUG	provider-helm	Observing	{"request": "vc1-8d6xs"}
2023-09-12T19:28:10.816Z	DEBUG	provider-helm	Updating	{"request": "vc1-8d6xs"}
2023-09-12T19:28:10.821Z	DEBUG	provider-helm	preparing upgrade for [vc1-vcluster]
2023-09-12T19:28:10.893Z	DEBUG	provider-helm	resetting values to the chart's original version%!(EXTRA []interface {}=[])

How can we reproduce it?

related release:

apiVersion: helm.crossplane.io/v1beta1
kind: Release
metadata:
  annotations:
    crossplane.io/composition-resource-name: vcluster-helm-release
    crossplane.io/external-create-pending: "2023-09-12T19:16:30Z"
    crossplane.io/external-create-succeeded: "2023-09-12T19:16:30Z"
    crossplane.io/external-name: vc1-vcluster
  creationTimestamp: "2023-09-12T19:16:30Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generateName: vc1-
  generation: 3
  labels:
    crossplane.io/claim-name: ""
    crossplane.io/claim-namespace: ""
    crossplane.io/composite: vc1
  name: vc1-8d6xs
  ownerReferences:
  - apiVersion: caas.telekom.de/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Vcluster
    name: vc1
    uid: 035ffe98-7a52-49d3-abe6-1b8b557f96ef
  resourceVersion: "72983650"
  uid: e7e9f382-6cb7-4e3c-bf5c-a5e28793bc76
spec:
  deletionPolicy: Delete
  forProvider:
    chart:
      name: vcluster
      repository: oci://mtr.devops.telekom.de/caas/charts
      url: oci://mtr.devops.telekom.de/caas/charts/vcluster:0.15.2
      version: 0.15.7
    namespace: vc1
    values:
      coredns:
        image: mtr.devops.telekom.de/rancher/mirrored-coredns-coredns:1.10.1
      defaultImageRegistry: mtr.devops.telekom.de
      storage:
        className: longhorn
      sync:
        hoststorageclasses:
          enabled: true
        ingresses:
          enabled: true
      syncer:
        image: /caas/vcluster:0.15.2
      vcluster:
        extraArgs:
        - --system-default-registry=mtr.devops.telekom.de
        image: /rancher/k3s:v1.26.0-k3s2-amd64
        resources:
          limits:
            cpu: 200m
            memory: 1Gi
          requests:
            cpu: 50m
            memory: 16Mi
  providerConfigRef:
    name: provider-helm
  rollbackLimit: 3
status:
  atProvider:
    releaseDescription: Upgrade complete
    revision: 999
    state: deployed
  conditions:
  - lastTransitionTime: "2023-09-12T19:16:32Z"
    reason: Unavailable
    status: "False"
    type: Ready
  - lastTransitionTime: "2023-09-12T19:16:31Z"
    reason: ReconcileSuccess
    status: "True"
    type: Synced

What environment did it happen in?

Crossplane version: v1.13.2 Crossplane-Helm-Version: v0.15.2 Kubernetes: v1.24.16

[eumel8]

correction: this occurs only in Helm Chart Vcluster Version 0.15.7 because the state is Unavailable

Downgrade 0.15.2 works, but hasn't the security features what we need. Still difficult to find out the root cause