Helm test hooks are not respected on retry

[steven-sheehy]

If an install or upgrade fails due to a failing helm test, upon retry it fails with an error that the test resource already exists. This causes a permanent failure of the helmrelease since even if the helm test would succeed during subsequent upgrades it is unable to run the helm test due to the existing resources.

This occurs despite having every possible helm test annotation on the test resources including before-hook-creation, hook-failed, and hook-succeeded. So helm-controller should've either removed the failed test resources when helm test failed or before the next install, but it did neither. If the helm test does succeed the test resources are successfully removed via hook-succeeded. The test resources are here and the deploy config is here.

helm-controller logs:

{"level":"error","ts":"2021-07-02T19:29:50.940Z","logger":"controller.helmrelease","msg":"unable to update status after reconciliation","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"mirror","namespace":"performance","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"mirror\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"error","ts":"2021-07-02T19:29:50.940Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"mirror","namespace":"performance","error":"Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io \"mirror\": the object has been modified; please apply your changes to the latest version and try again"}
{"level":"info","ts":"2021-07-02T19:29:50.974Z","logger":"controller.helmrelease","msg":"all dependencies are ready, proceeding with release","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"mirror","namespace":"performance"}
2021/07/02 19:29:51 warning: cannot overwrite table with non table for HEDERA_MIRROR_REST_DB_HOST (map[valueFrom:map[secretKeyRef:map[key:HEDERA_MIRROR_IMPORTER_DB_HOST name:mirror-passwords]]])
2021/07/02 19:29:51 warning: cannot overwrite table with non table for HEDERA_MIRROR_REST_DB_HOST (map[valueFrom:map[secretKeyRef:map[key:HEDERA_MIRROR_IMPORTER_DB_HOST name:mirror-passwords]]])
{"level":"info","ts":"2021-07-02T19:29:56.321Z","logger":"controller.helmrelease","msg":"all dependencies are ready, proceeding with release","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"mirror","namespace":"integration"}
{"level":"info","ts":"2021-07-02T19:29:56.407Z","logger":"controller.helmrelease","msg":"reconcilation finished in 86.305884ms, next run in 30s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"mirror","namespace":"integration"}
{"level":"info","ts":"2021-07-02T19:29:57.384Z","logger":"controller.helmrelease","msg":"reconcilation finished in 6.443523264s, next run in 1m30s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"mirror","namespace":"performance"}
{"level":"error","ts":"2021-07-02T19:29:57.384Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"mirror","namespace":"performance","error":"Helm test failed: warning: Hook test hedera-mirror/charts/rest/templates/tests/pod.yaml failed: object is being deleted: pods \"mirror-rest-test\" already exists"}
{"level":"info","ts":"2021-07-02T19:29:57.385Z","logger":"controller.helmrelease","msg":"all dependencies are ready, proceeding with release","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"mirror","namespace":"performance"}
2021/07/02 19:29:57 warning: cannot overwrite table with non table for HEDERA_MIRROR_REST_DB_HOST (map[valueFrom:map[secretKeyRef:map[key:HEDERA_MIRROR_IMPORTER_DB_HOST name:mirror-passwords]]])
2021/07/02 19:29:57 warning: cannot overwrite table with non table for HEDERA_MIRROR_REST_DB_HOST (map[valueFrom:map[secretKeyRef:map[key:HEDERA_MIRROR_IMPORTER_DB_HOST name:mirror-passwords]]])
{"level":"info","ts":"2021-07-02T19:30:04.384Z","logger":"controller.helmrelease","msg":"reconcilation finished in 6.999359776s, next run in 1m30s","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"mirror","namespace":"performance"}
{"level":"error","ts":"2021-07-02T19:30:04.384Z","logger":"controller.helmrelease","msg":"Reconciler error","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"mirror","namespace":"performance","error":"Helm test failed: warning: Hook test hedera-mirror/charts/rest/templates/tests/pod.yaml failed: object is being deleted: pods \"mirror-rest-test\" already exists"}
{"level":"info","ts":"2021-07-02T19:30:04.385Z","logger":"controller.helmrelease","msg":"all dependencies are ready, proceeding with release","reconciler group":"helm.toolkit.fluxcd.io","reconciler kind":"HelmRelease","name":"mirror","namespace":"performance"}
2021/07/02 19:30:04 warning: cannot overwrite table with non table for HEDERA_MIRROR_REST_DB_HOST (map[valueFrom:map[secretKeyRef:map[key:HEDERA_MIRROR_IMPORTER_DB_HOST name:mirror-passwords]]])
2021/07/02 19:30:04 warning: cannot overwrite table with non table for HEDERA_MIRROR_REST_DB_HOST (map[valueFrom:map[secretKeyRef:map[key:HEDERA_MIRROR_IMPORTER_DB_HOST name:mirror-passwords]]])

[steven-sheehy]

I also get that the object has been modified; please apply your changes to the latest version and try again error quite often but it usually retries and resolves itself. Note I did not modify the helmrelease manually as it indicates. In fact, I had deleted the namespace and this was a brand new install.

[kingdonb]

Thanks for the report, I'll give reproducing this issue a shot soon so I can better understand it.

[steven-sheehy]

Thanks @kingdonb. I think the other issue about object has been modified I mention in a comment might be due to https://github.com/fluxcd/helm-controller/issues/295 and is probably not related to this issue, but not sure.