Unable to delete Cluster Image Policy after uninstalling Policy Controller Helm chart

[oscarbetancurj]

Description

When you deploy policy controller helm chart and then you apply a CIP, if you uninstall the helm chart without deleting first the CIP, you will be unable to delete the CIP. Error message:

"Error from server: conversion webhook for policy.sigstore.dev/v1alpha1, Kind=ClusterImagePolicy failed: Post "https://policy-webhook.cosign-system.svc:443/resource-conversion?timeout=30s": service "policy-webhook" not found"

Steps to reproduce:

• Deploy policy controller helm chart:

helm repo add sigstore https://sigstore.github.io/helm-charts helm repo update kubectl create namespace cosign-system helm install policy-controller -n cosign-system sigstore/policy-controller

• Apply CIP (Cluster Image Policy)

kubectl apply -f cip.yaml

apiVersion: policy.sigstore.dev/v1alpha1 kind: ClusterImagePolicy metadata: name: custom-key spec: images:

• glob: "**" authorities:

• name: kms-key key: kms: awskms:///alias/cosign-test-key

• Uninstall helm chart:

helm uninstall policy-controller -n cosign-system

• Try to delete the CIP:

kubectl delete clusterimagepolicies.policy.sigstore.dev custom-key

Error: "Error from server: conversion webhook for policy.sigstore.dev/v1alpha1, Kind=ClusterImagePolicy failed: Post "https://policy-webhook.cosign-system.svc:443/resource-conversion?timeout=30s": service "policy-webhook" not found"

Version

All versions tested (0.2.3, 0.3.0, 0.3.6)

[hectorj2f]

@oscarbetancurj I have tested it and I didn't manage to reproduce your problem. Perhaps there was some leftovers from a previous failed uninstallation.

➜ helm install policy-controller -n cosign-system sigstore/policy-controller
NAME: policy-controller
LAST DEPLOYED: Thu Dec  1 10:37:31 2022
NAMESPACE: cosign-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

➜   helm ls  -n cosign-system
NAME             	NAMESPACE    	REVISION	UPDATED                             	STATUS  	CHART                  	APP VERSION
policy-controller	cosign-system	1       	2022-12-01 10:37:31.396439 +0100 CET	deployed	policy-controller-0.3.8	0.4.2
➜   k get validatingwebhookconfiguration -A
NAME                                         WEBHOOKS   AGE
policy.sigstore.dev                          1          31s
validating.clusterimagepolicy.sigstore.dev   1          31s
➜  k get mutatingwebhookconfiguration -A
NAME                                         WEBHOOKS   AGE
defaulting.clusterimagepolicy.sigstore.dev   1          43s
policy.sigstore.dev                          1          43s
➜  k create -f test/testdata/policy-controller/tester/cip-public-keyless.yaml
Warning: missing field(s): spec.authorities[0].keyless.identities
clusterimagepolicy.policy.sigstore.dev/demo created
➜   k get cip -A
NAME   AGE
demo   4s
➜   helm uninstall policy-controller -n cosign-system

release "policy-controller" uninstalled
➜   k get mutatingwebhookconfiguration -A
No resources found
➜   k get validatingwebhookconfiguration -A
No resources found
➜   k get cip -A
Error from server: conversion webhook for policy.sigstore.dev/v1alpha1, Kind=ClusterImagePolicy failed: Post "https://policy-webhook.cosign-system.svc:443/resource-conversion?timeout=30s": service "policy-webhook" not found

[hectorj2f]

I am using helm v3.10.0, if you still face this issue we could try adding a cleanup job to ensure the webhooks are deleted.

[oscarbetancurj]

@hectorj2f Hi!! But this error message: "k get cip -A Error from server: conversion webhook for policy.sigstore.dev/v1alpha1, Kind=ClusterImagePolicy failed: Post "https://policy-webhook.cosign-system.svc:443/resource-conversion?timeout=30s": service "policy-webhook" not found"

It is the same I got in all namespaces.

If you try to delete the CIP (It is still present), can you do it? That's my problem :)

[gustavoromerobenitez]

I had the same issue and solved it deleting and re-creating the CRD:

kubectl delete crd clusterimagepolicies.policy.sigstore.dev

[oscarbetancurj]

Thanks @gustavoromerobenitez. I did the same and I could solve the issue. I had to patch the CRD like this, because it got deleting for too much time:

kubectl patch crd clusterimagepolicies.policy.sigstore.dev -p '{"metadata":{"finalizers":[]}}' --type=merge

[hectorj2f]

The helm deletion doesn't check if there are policies left in the system.

[hectorj2f]

Maybe @cpanato knows any other way via helm.