Not able to add custom plugin in kubernetes version 1.24.9 Works fine in 1.23

[shreyasHpandya]

When I try to create plugin it throws following error

Error from server: error when creating "custom-plugin.yaml": admission webhook "validations.kong.konghq.com" denied the request: plugin failed schema validation: schema violation (name: plugin 'custom-plugin' not enabled; add it to the 'plugins' configuration property)

in values.yaml i have set the plugin properly

ingressController:
  installCRDs: false
  ingressClass: kong
plugins:
  configMaps:
    - name: kong-plugin-custom-plugin
      pluginName: custom-plugin
proxy:
  loadBalancerIP: <ip>
podAnnotations:
   prometheus.io/scrape: "true"
   prometheus.io/port: "8100"

The config map kong-plugin-custom-plugin is also created as per custom plugin docs before applying kong helm chart.

For some reason this does not work in kubernetes version 1.24.9. Same setup works fine in 1.23

UPDATE Found the root cause. If you have multiple kong installed in same cluster only one of the Admission-Webhook will be active form either of the helm chart. And if the custom plugin you are trying to install is configured in the Kong which is not the one referred to by the Admission-Webhook, then webhook rejects the plugin installation since the Kong instance it is calling has no knowledge of it.

[snxq]

@shreyasHpandya I have the same error.Is there any way to circumvent this restriction?

[rainest]

There's not any particularly straightforward way to deal with these. The easiest option is probably using the same set of installed custom plugins on all Kong instances within a cluster.

The more complicated approach would be to limit the admission webhook scope, by adding additional label filters to the various rules, and then labeling your plugin resources accordingly to direct them to a particular instance's webhook.

We wouldn't want that as a default requirement on admission for our CRDs, and adding it as an optional chart feature would be fairly involved, so for the time being you'd need to edit the admission rules manually and rely on Helm's 3-way diff logic to leave those edits in place.