k3s

k3s copied to clipboard

When a CRD contains an OpenAPIV3Schema validation spec, the validation is not executed in k3s, and invalid resources are accepted by the API Server.

To Reproduce test-crd.yaml:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: examplecrds.example.com
spec:
  scope: Namespaced
  group: example.com
  versions:
  - name: v1
    served: true
    storage: true
  names:
    kind: ExampleCRD
    plural: examplecrds
    singular: examplecrd
  validation:
    openAPIV3Schema:
      required: [spec]
      properties:
        spec:
          required: [musthave]
          properties:
            musthave:
              enum: [one,two,three]
            irrelevant:
              enum: [something]

valid-resource.yaml:

apiVersion: example.com/v1
kind: ExampleCRD
metadata:
  name: test
spec:
  musthave: two

invalid-resources.yaml:

---
apiVersion: example.com/v1
kind: ExampleCRD
metadata:
  name: invalid1
spec:
  musthave: four
---
apiVersion: example.com/v1
kind: ExampleCRD
metadata:
  name: invalid2
spec:
  musthave: three
  irrelevant: bad
---
apiVersion: example.com/v1
kind: ExampleCRD
metadata:
  name: invalid3
spec:
  irrelevant: something

Expected behavior

➜ kubectl apply -f test-crd.yaml
customresourcedefinition.apiextensions.k8s.io/examplecrds.example.com created

➜ kubectl apply -f valid-resource.yaml
examplecrd.example.com/test created

➜ kubectl apply -f invalid-resources.yaml
Error from server (Invalid): error when creating "invalid-resources.yaml": ExampleCRD.example.com "invalid1" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"example.com/v1", "kind":"ExampleCRD", "metadata":map[string]interface {}{"name":"invalid1", "namespace":"db", "creationTimestamp":"2019-05-03T14:05:10Z", "annotations":map[string]interface {}{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"example.com/v1\",\"kind\":\"ExampleCRD\",\"metadata\":{\"annotations\":{},\"name\":\"invalid1\",\"namespace\":\"db\"},\"spec\":{\"musthave\":\"four\"}}\n"}, "generation":1, "uid":"76e5ce1d-6dac-11e9-9968-021ac63aea36"}, "spec":map[string]interface {}{"musthave":"four"}}: validation failure list:
spec.musthave in body should be one of [one two three]
Error from server (Invalid): error when creating "invalid-resources.yaml": ExampleCRD.example.com "invalid2" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"example.com/v1", "kind":"ExampleCRD", "metadata":map[string]interface {}{"generation":1, "uid":"77403101-6dac-11e9-9968-021ac63aea36", "name":"invalid2", "namespace":"db", "creationTimestamp":"2019-05-03T14:05:11Z", "annotations":map[string]interface {}{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"example.com/v1\",\"kind\":\"ExampleCRD\",\"metadata\":{\"annotations\":{},\"name\":\"invalid2\",\"namespace\":\"db\"},\"spec\":{\"irrelevant\":\"bad\",\"musthave\":\"three\"}}\n"}}, "spec":map[string]interface {}{"irrelevant":"bad", "musthave":"three"}}: validation failure list:
spec.irrelevant in body should be one of [something]
Error from server (Invalid): error when creating "invalid-resources.yaml": ExampleCRD.example.com "invalid3" is invalid: []: Invalid value: map[string]interface {}{"kind":"ExampleCRD", "metadata":map[string]interface {}{"creationTimestamp":"2019-05-03T14:05:11Z", "annotations":map[string]interface {}{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"example.com/v1\",\"kind\":\"ExampleCRD\",\"metadata\":{\"annotations\":{},\"name\":\"invalid3\",\"namespace\":\"db\"},\"spec\":{\"irrelevant\":\"something\"}}\n"}, "generation":1, "uid":"77982fb8-6dac-11e9-9968-021ac63aea36", "name":"invalid3", "namespace":"db"}, "spec":map[string]interface {}{"irrelevant":"something"}, "apiVersion":"example.com/v1"}: validation failure list:
spec.musthave in body is required

Actual Output

➜ kubectl apply -f test-crd.yaml
customresourcedefinition.apiextensions.k8s.io/examplecrds.example.com created

➜ kubectl apply -f valid-resource.yaml
examplecrd.example.com/test created

➜ kubectl apply -f invalid-resources.yaml
examplecrd.example.com/invalid1 created
examplecrd.example.com/invalid2 created
examplecrd.example.com/invalid3 created

OpenAPI support does not exist in k3s atm. OpenAPI adds a huge amount of bloat so we are working on a solution here. Right now the openapi schema that k3s serves up is a static file.

CRD validation and serving openapi for aggregated APIs are the known broken areas for k3s. K8s 1.15 will introduce more features that require openapi so we are working on this.

Good to know. Jusy started experimenting with this part of CRD-s, so it's no biggie (not a deal breaker). I am playing with k3s for local development and I tend to do k8s experiments mainly with k3s (fastest way to bootstrap a cluster I know of). But I guess this could surprise some people who rely on it more than I do and I want k3s to succeed. :)

Using apiextensions.k8s.io/v1 crds instead of v1beta1 might resolve this. I'd noticed invalid extra fields weren't being rejected in k3s for my v1beta1 crd. I started seeing expected rejections after switching to v1 (in order to use CustomResourceDefaulting)

I was able to resolve it for yamls after referring to this example: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#specify-multiple-versions and this guide: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#customresourcedefinition-v122

Closing due to age