[Bug] Variables in message are not getting evaluated before pushing to webhook/ui

[franznemeth]

We are using policy-reporter to push policy-violations to a service which then handles the further processing and making sure the owner of the violating resource is informed about the violation.

For this we are adapting our kyverno policies to include some metadata in the violation message. This has always worked fine in the past (kyverno chart version: 2.7.0, kyverno version: v1.9.0 and policy-reporter chart version: 2.13.0)

After upgrading to the latest versions for kyverno and policy-reporter this functionality no longer works.

Policy-reporter pushes the message as is to the configured endpoint without first replacing the variable in {{}}.

Here is a policy to test this with:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  annotations:
    policies.kyverno.io/severity: high
    policies.kyverno.io/title: Disallow host namespaces
  name: disallow-host-namespaces
spec:
  admission: true
  background: false
  failurePolicy: Fail
  rules:
  - exclude:
      resources: {}
    generate:
      clone: {}
      cloneList: {}
    match:
      resources:
        kinds:
        - Pod
    mutate: {}
    name: host-namespaces
    validate:
      message: |
        Sharing the host namespaces is disallowed. The fields spec.hostNetwork, spec.hostIPC, and spec.hostPID must not be set to true. test:{{request.object.metadata.annotations.test||'none'}}
      pattern:
        spec:
          =(hostIPC): "false"
          =(hostNetwork): "false"
          =(hostPID): "false"
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 15

and the following pod to test the validation:

apiVersion: v1
kind: Pod
metadata:
  name: test
  labels:
    app: test
  annotations:
    test: https://example.com/some-url/1
spec:
  hostNetwork: true
  containers:
    - name: bad-hostnetwork
      image: docker.io/library/nginx
      imagePullPolicy: IfNotPresent
  restartPolicy: Always

Applying this pod against a cluster where kyverno is installed gives the following message:

kubectl apply -f test.yaml -n default
Error from server: error when creating "disallow-host-namespaces/host-namespaces/fail.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Pod/default/bad-hostnetwork was blocked due to the following policies

disallow-host-namespaces:
  host-namespaces: |-
    validation error: Sharing the host namespaces is disallowed. The fields spec.hostNetwork, spec.hostIPC, and spec.hostPID must not be set to true. test:https://example.com/some-url/1
    . rule host-namespaces failed at path /spec/hostNetwork/

While the push to the endpoint looks like this:

{"message":"Sharing the host namespaces is disallowed. The fields spec.hostNetwork, spec.hostIPC, and spec.hostPID must not be set to true. test:{{request.object.metadata.annotations.test||'none'}}\n","policy":"disallow-host-namespaces","rule":"host-namespaces","priority":"error","status":"fail","severity":"high","category":"Pod Security Standards (Default)","scored":false,"properties":{"eventName":"disallow-host-namespaces.17a28fa7ec924b5e","resultID":"33991761c902176dd731f6f5b4a88822d330874d","time":"0001-01-01T00:00:00Z"},"resource":{"apiVersion":"","kind":"Pod","name":"test","namespace":"default","uid":""},"creationTimestamp":"0001-01-01T00:00:00Z","source":"Kyverno Event"}

[fjogeleit]

Hey, I have a look. At the first view I think this is more related to Kyverno, which is responsible for the evalutation. Policy Reporter just use the evailable text from the reports.

In case of the blockReport feature of the plugin, it uses the K8s events as source, which is also produced by Kyverno.

cc @eddycharly @realshuting do you know recent changes which could lead to this issue?

[franznemeth]

Hi, is there anything I could do to help this get resolved. Should I be looking at the policy-reporter code or kyverno? Thanks

[fjogeleit]

hey, I reached out to @eddycharly, it will be most likely an issue in Kyverno because Policy Reporter just shows information from Kyverno / Policy Reports / Events and does no Variables evaluation on its own.

[realshuting]

I tested this policy in Audit mode and the variable was resolved correctly in the policyreport, I'm using v1.12.0-alpha.2:

✗ k get polr -o yaml | grep message -A2
  - message: |-
      validation error: Sharing the host namespaces is disallowed. The fields spec.hostNetwork, spec.hostIPC, and spec.hostPID must not be set to true. test:https://example.com/some-url/1
      . rule host-namespaces failed at path /spec/hostNetwork/

@franznemeth - can you help verify with the latest version?

[franznemeth]

Hi, sadly I'm still facing the same issue on v1.12.0-alpha.2.

We managed to fix the issue though and it had nothing to do with kyverno itself.

In our policy-reporter values.yaml we had the following things set:

replicaCount: 1
kyvernoPlugin:
  enabled: true
global:
  plugins:
    kyverno: true

Leaving both keys on false (which is the default) fixes the webhook push and the template gets rendered successfully.

Any ideas why these settings would break the resolving of those variables?

[fjogeleit]

Okay, I think I got it, we are talking about PolicyReports that are not generated from Kyverno, we talk about PolicyReports which are created in the Policy Reporter Kyverno Plugin (https://github.com/kyverno/policy-reporter-kyverno-plugin/blob/main/pkg/policyreport/kubernetes/client.go)

Which creates the message from the Kubernetes event message. If you deactivate the plugin, these PolicyReports will of course no longer be created.

So I assume that the variables might not be resolved for blocked resources in the kubernetes event messages @realshuting

[realshuting]

Which creates the message from the Kubernetes event message. If you deactivate the plugin, these PolicyReports will of course no longer be created.

So I assume that the variables might not be resolved for blocked resources in the kubernetes event messages @realshuting

Can we warn in this case if Kyverno is not enabled?

[fjogeleit]

What do you mean with Kyverno not enabled?

Its basically:

• The Policy is in Enforce mode
• The User tries to create a failing resource
• The Resource is blocked and a Kubernetes Event created
• Policy Reporter Kyverno Plugin picks the K8s Event up and creates a PolicyReport out of it

So Kyverno is enabled and working but doesn't resolve the variables in the K8s Event Message but in case of Audit or Background Scan in the PolicyReports (created by Kyverno)