kubeless
kubeless copied to clipboard
vmware-archive
Starting kubeless | kubeless-controller-manager in CrashLoopBackOff
Is this a BUG REPORT or FEATURE REQUEST?: BUG
What happened: Installed kubeless with below
kube-shell> kubectl create ns kubeless
namespace/kubeless created
kube-shell> kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml
configmap/kubeless-config created
deployment.apps/kubeless-controller-manager created
serviceaccount/controller-acct created
customresourcedefinition.apiextensions.k8s.io/functions.kubeless.io created
customresourcedefinition.apiextensions.k8s.io/httptriggers.kubeless.io created
customresourcedefinition.apiextensions.k8s.io/cronjobtriggers.kubeless.io created
kube-shell> kubectl --namespace kubeless get pods
NAME READY STATUS RESTARTS AGE
kubeless-controller-manager-67d484f5d-xf2mj 0/3 CrashLoopBackOff 9 107s
What you expected to happen: Kubeless Pods being created
How to reproduce it (as minimally and precisely as possible):
kubectl create ns kubeless
kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version):
Client Version: v1.16.6-beta.0
Server Version: v1.18.3
- Kubeless version (use
kubeless version): v1.0.7-dirty - Cloud provider or physical cluster: Local Mac
Pod description :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m25s default-scheduler Successfully assigned kubeless/kubeless-controller-manager-67d484f5d-xf2mj to minikube
Normal Created 2m22s (x2 over 2m23s) kubelet, minikube Created container http-trigger-controller
Normal Created 2m22s (x2 over 2m23s) kubelet, minikube Created container kubeless-function-controller
Normal Started 2m22s (x2 over 2m23s) kubelet, minikube Started container kubeless-function-controller
Normal Pulled 2m22s (x2 over 2m23s) kubelet, minikube Container image "kubeless/http-trigger-controller:v1.0.1" already present on machine
Normal Started 2m21s (x2 over 2m23s) kubelet, minikube Started container http-trigger-controller
Normal Started 2m21s (x2 over 2m22s) kubelet, minikube Started container cronjob-trigger-controller
Normal Pulled 2m21s (x2 over 2m23s) kubelet, minikube Container image "kubeless/cronjob-trigger-controller:v1.0.1" already present on machine
Normal Created 2m21s (x2 over 2m23s) kubelet, minikube Created container cronjob-trigger-controller
Warning BackOff 2m20s (x2 over 2m21s) kubelet, minikube Back-off restarting failed container
Warning BackOff 2m20s (x2 over 2m21s) kubelet, minikube Back-off restarting failed container
Warning BackOff 2m20s (x2 over 2m21s) kubelet, minikube Back-off restarting failed container
Normal Pulled 2m8s (x3 over 2m23s) kubelet, minikube Container image "kubeless/function-controller:v1.0.7" already present on machine
I @iamshreeram, can you get the logs of the container failing? First you need to get which one of the 3 containers is failing (you can that info from the pod description), then get the logs with kubectl logs -n kubeless kubeless-controller-manager-67d484f5d-xf2mj -c <container>. That should give more info about what's gone wrong.
@andresmgot , Thanks for looking at the issue.
Below is the error for all 3 containers [kubeless-function-controller, http-trigger-controller, cronjob-trigger-controller] in all 3 pods :
time="2020-07-20T12:17:04Z" level=info msg="Running Kubeless controller manager version: v1.0.7"
time="2020-07-20T12:17:04Z" level=fatal msg="Unable to read the configmap: Error while fetching config location: customresourcedefinitions.apiextensions.k8s.io \"functions.kubeless.io\" is forbidden: User \"system:serviceaccount:kubeless:controller-acct\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope"
ah, I see the problem. You are using the manifest that doesn't include the RBAC configuration (and RBAC is now always enabled by default). Install Kubeless like this:
kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-v1.0.7.yaml
(rather than using the non-rbac version)
@andresmgot , Thanks for response.
Does this mean users can never use non-rbac - kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml? If so, can this be removed from documentation?
While it's not usual, people can disable RBAC in their clusters. If they do so, the manifest with RBAC rules will fail, that's why we offer the two options. In the documentation, we point to the other manifest but if you think of another way of making it more prominent please send a PR to help others:
https://github.com/kubeless/kubeless/blob/master/docs/quick-start.md
@andresmgot , I think we can just remove - kubeless-non-rbac-$RELEASE.yaml is used for non-RBAC Kubernetes cluster. from the doc and provide documentation to disable RBAC.
Let me know. thanks!
I had the same issue, and it took me a while until I found this page. I also vote for the above proposal.
I am having the same issue but not due to RBAC
kubeless-controller-manager-5d9bf7f5c6-hs7s5 2/3 CrashLoopBackOff 5 3m30s
The cronjob-trigger-controller container is crashing. Following with following the logs
time="2021-05-02T20:30:09Z" level=info msg="Running Kubeless cronjob trigger controller version: v1.0.3"
time="2021-05-02T20:30:09Z" level=fatal msg="Unable to read the configmap: Error while fetching config location: v1beta1.CustomResourceDefinition: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 535 ...s\":{\"k:{\\\"... at {\"kind\":\"CustomResourceDefinition\",\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"metadata\":{\"name\":\"functions.kubeless.io\",\"uid\":\"1fb634d2-032e-472b-a8c7-10edd69e96f8\",\"resourceVersion\":\"582946\",\"generation\":1,\"creationTimestamp\":\"2021-05-02T20:29:48Z\",\"managedFields\":[{\"manager\":\"kube-apiserver\",\"operation\":\"Update\",\"apiVersion\":\"apiextensions.k8s.io/v1\",\"time\":\"2021-05-02T20:29:48Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:status\":{\"f:acceptedNames\":{\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:conditions\":{\"k:{\\\"type\\\":\\\"Established\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NamesAccepted\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NonStructuralSchema\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}}}}},{\"manager\":\"kubectl-create\",\"operation\":\"Update\",\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"time\":\"2021-05-02T20:29:48Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:spec\":{\"f:conversion\":{\".\":{},\"f:strategy\":{}},\"f:group\":{},\"f:names\":{\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:preserveUnknownFields\":{},\"f:scope\":{},\"f:version\":{},\"f:versions\":{}}}}]},\"spec\":{\"group\":\"kubeless.io\",\"version\":\"v1beta1\",\"names\":{\"plural\":\"functions\",\"singular\":\"function\",\"kind\":\"Function\",\"listKind\":\"FunctionList\"},\"scope\":\"Namespaced\",\"versions\":[{\"name\":\"v1beta1\",\"served\":true,\"storage\":true}],\"conversion\":{\"strategy\":\"None\"},\"preserveUnknownFields\":true},\"status\":{\"conditions\":[{\"type\":\"NamesAccepted\",\"status\":\"True\",\"lastTransitionTime\":\"2021-05-02T20:29:48Z\",\"reason\":\"NoConflicts\",\"message\":\"no conflicts found\"},{\"type\":\"Established\",\"status\":\"True\",\"lastTransitionTime\":\"2021-05-02T20:29:48Z\",\"reason\":\"InitialNamesAccepted\",\"message\":\"the initial names have been accepted\"},{\"type\":\"NonStructuralSchema\",\"status\":\"True\",\"lastTransitionTime\":\"2021-05-02T20:29:48Z\",\"reason\":\"Violations\",\"message\":\"spec.preserveUnknownFields: Invalid value: true: must be false\"}],\"acceptedNames\":{\"plural\":\"functions\",\"singular\":\"function\",\"kind\":\"Function\",\"listKind\":\"FunctionList\"},\"storedVersions\":[\"v1beta1\"]}}"
@shubhomoy Kubernetes +1.18 is not supported for the cronjob trigger so that may be the reason. See #1130
I got the same error as @shubhomoy on:
kubeless version
Kubeless version: v1.0.8
kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
The error (parsing position is slightly different, but the error is the same):
kubectl logs kubeless-controller-manager-5d9bf7f5c6-sqhvg -n kubeless -c cronjob-trigger-controller
time="2021-07-11T19:53:44Z" level=info msg="Running Kubeless cronjob trigger controller version: v1.0.3"
time="2021-07-11T19:53:44Z" level=fatal msg="Unable to read the configmap: Error while fetching config location: v1beta1.CustomResourceDefinition: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 901 ...s\":{\"k:{\\\"... at {\"kind\":\"CustomResourceDefinition\",\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"metadata\":{\"name\":\"functions.kubeless.io\",\"uid\":\"f0b89e5a-0ca7-4db2-b133-f65a50fc2566\",\"resourceVersion\":\"2620\",\"generation\":1,\"creationTimestamp\":\"2021-07-11T19:36:54Z\",\"managedFields\":[{\"manager\":\"kubectl-create\",\"operation\":\"Update\",\"apiVersion\":\"apiextensions.k8s.io/v1beta1\",\"time\":\"2021-07-11T19:36:54Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:spec\":{\"f:conversion\":{\".\":{},\"f:strategy\":{}},\"f:group\":{},\"f:names\":{\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:preserveUnknownFields\":{},\"f:scope\":{},\"f:version\":{},\"f:versions\":{}}}},{\"manager\":\"kube-apiserver\",\"operation\":\"Update\",\"apiVersion\":\"apiextensions.k8s.io/v1\",\"time\":\"2021-07-11T19:36:55Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:status\":{\"f:acceptedNames\":{\"f:kind\":{},\"f:listKind\":{},\"f:plural\":{},\"f:singular\":{}},\"f:conditions\":{\"k:{\\\"type\\\":\\\"Established\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NamesAccepted\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"NonStructuralSchema\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}}}}}]},\"spec\":{\"group\":\"kubeless.io\",\"version\":\"v1beta1\",\"names\":{\"plural\":\"functions\",\"singular\":\"function\",\"kind\":\"Function\",\"listKind\":\"FunctionList\"},\"scope\":\"Namespaced\",\"versions\":[{\"name\":\"v1beta1\",\"served\":true,\"storage\":true}],\"conversion\":{\"strategy\":\"None\"},\"preserveUnknownFields\":true},\"status\":{\"conditions\":[{\"type\":\"NonStructuralSchema\",\"status\":\"True\",\"lastTransitionTime\":\"2021-07-11T19:36:54Z\",\"reason\":\"Violations\",\"message\":\"spec.preserveUnknownFields: Invalid value: true: must be false\"},{\"type\":\"NamesAccepted\",\"status\":\"True\",\"lastTransitionTime\":\"2021-07-11T19:36:54Z\",\"reason\":\"NoConflicts\",\"message\":\"no conflicts found\"},{\"type\":\"Established\",\"status\":\"True\",\"lastTransitionTime\":\"2021-07-11T19:36:55Z\",\"reason\":\"InitialNamesAccepted\",\"message\":\"the initial names have been accepted\"}],\"acceptedNames\":{\"plural\":\"functions\",\"singular\":\"function\",\"kind\":\"Function\",\"listKind\":\"FunctionList\"},\"storedVersions\":[\"v1beta1\"]}}"
Error pretty print
{
"kind":"CustomResourceDefinition",
"apiVersion":"apiextensions.k8s.io/v1beta1",
"metadata":{
"name":"functions.kubeless.io",
"uid":"f0b89e5a-0ca7-4db2-b133-f65a50fc2566",
"resourceVersion":"2620",
"generation":1,
"creationTimestamp":"2021-07-11T19:36:54Z",
"managedFields":[
{
"manager":"kubectl-create",
"operation":"Update",
"apiVersion":"apiextensions.k8s.io/v1beta1",
"time":"2021-07-11T19:36:54Z",
"fieldsType":"FieldsV1",
"fieldsV1":{
"f:spec":{
"f:conversion":{
".":{
},
"f:strategy":{
}
},
"f:group":{
},
"f:names":{
"f:kind":{
},
"f:listKind":{
},
"f:plural":{
},
"f:singular":{
}
},
"f:preserveUnknownFields":{
},
"f:scope":{
},
"f:version":{
},
"f:versions":{
}
}
}
},
{
"manager":"kube-apiserver",
"operation":"Update",
"apiVersion":"apiextensions.k8s.io/v1",
"time":"2021-07-11T19:36:55Z",
"fieldsType":"FieldsV1",
"fieldsV1":{
"f:status":{
"f:acceptedNames":{
"f:kind":{
},
"f:listKind":{
},
"f:plural":{
},
"f:singular":{
}
},
"f:conditions":{
"k:{""type"":""Established""}":{
".":{
},
"f:lastTransitionTime":{
},
"f:message":{
},
"f:reason":{
},
"f:status":{
},
"f:type":{
}
},
"k:{""type"":""NamesAccepted""}":{
".":{
},
"f:lastTransitionTime":{
},
"f:message":{
},
"f:reason":{
},
"f:status":{
},
"f:type":{
}
},
"k:{""type"":""NonStructuralSchema""}":{
".":{
},
"f:lastTransitionTime":{
},
"f:message":{
},
"f:reason":{
},
"f:status":{
},
"f:type":{
}
}
}
}
}
}
]
},
"spec":{
"group":"kubeless.io",
"version":"v1beta1",
"names":{
"plural":"functions",
"singular":"function",
"kind":"Function",
"listKind":"FunctionList"
},
"scope":"Namespaced",
"versions":[
{
"name":"v1beta1",
"served":true,
"storage":true
}
],
"conversion":{
"strategy":"None"
},
"preserveUnknownFields":true
},
"status":{
"conditions":[
{
"type":"NonStructuralSchema",
"status":"True",
"lastTransitionTime":"2021-07-11T19:36:54Z",
"reason":"Violations",
"message":"spec.preserveUnknownFields: Invalid value: true: must be false"
},
{
"type":"NamesAccepted",
"status":"True",
"lastTransitionTime":"2021-07-11T19:36:54Z",
"reason":"NoConflicts",
"message":"no conflicts found"
},
{
"type":"Established",
"status":"True",
"lastTransitionTime":"2021-07-11T19:36:55Z",
"reason":"InitialNamesAccepted",
"message":"the initial names have been accepted"
}
],
"acceptedNames":{
"plural":"functions",
"singular":"function",
"kind":"Function",
"listKind":"FunctionList"
},
"storedVersions":[
"v1beta1"
]
}
}
I have tried to add preserveUnknownFields: false at the end of kubeless-v1.0.8.yaml:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cronjobtriggers.kubeless.io
spec:
group: kubeless.io
names:
kind: CronJobTrigger
plural: cronjobtriggers
singular: cronjobtrigger
scope: Namespaced
version: v1beta1
preserveUnknownFields: false
But it causes:
The CustomResourceDefinition "cronjobtriggers.kubeless.io" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: because otherwise all fields are pruned
Which makes sense. Can someone please point me, where are the object which openAPIV3Schema must describe is defined/located? I can write a schema from that and try to run kubeless with that.
Another question - can cronjobtriggers be safely removed? I'm only interested in http and cli triggers.
Links:
- Similar issue
- https://github.com/kubernetes/apiextensions-apiserver/issues/25
Trivial fix (preserveUnknownFields: false) didn't work, the same error persists.
New yaml for cronjobtriggers.kubeless.io
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cronjobtriggers.kubeless.io
spec:
group: kubeless.io
names:
kind: CronJobTrigger
plural: cronjobtriggers
singular: cronjobtrigger
scope: Namespaced
version: v1beta1
versions:
- name: v1beta1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
schedule:
type: string
jobTemplate:
type: object
- name: v1
served: false
storage: false
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
schedule:
type: string
jobTemplateA:
type: object
preserveUnknownFields: false
@eduard93 @eduard93 @shubhomoy @iamshreeram the reason of the crashbackoff is cronjob-trigger-controller version v1.0.3 dependence kubeless version define as github.com/kubeless/kubeless v1.0.0-alpha.6, the kubeless v1.0.0-alpha.6 dependece client-go version was very low (k8s.io/client-go v5.0.0+incompatible), the decode interface not compatible the high version kubernetes crd resource.
how to solve this problem？
rebuild the cronjob-trigger-controller: git clone https://github.com/kubeless/cronjob-trigger.git && cd cronjob-trigger, and the modify the go.mod, change the github.com/kubeless/kubeless v1.0.0-alpha.6 to github.com/kubeless/kubeless v1.0.8, the v1.0.8 is the version which you to use kubeless’s version , and the go mod tidy , rebuild the cronjob-trigger-controller image, that will be succedded.
@eduard93 @eduard93 @shubhomoy @iamshreeram the reason of the crashbackoff is cronjob-trigger-controller version v1.0.3 dependence kubeless version define as github.com/kubeless/kubeless v1.0.0-alpha.6, the kubeless v1.0.0-alpha.6 dependece client-go version was very low (k8s.io/client-go v5.0.0+incompatible), the decode interface not compatible the high version kubernetes crd resource.
how to solve this problem？
rebuild the cronjob-trigger-controller: git clone https://github.com/kubeless/cronjob-trigger.git && cd cronjob-trigger, and the modify the go.mod, change the github.com/kubeless/kubeless v1.0.0-alpha.6 to github.com/kubeless/kubeless v1.0.8, the v1.0.8 is the version which you to use kubeless’s version , and the go mod tidy , rebuild the cronjob-trigger-controller image, that will be succedded.
Did you manage to work kubeless with this modification on k8s > 1.18? I am trying to move our clusters to 1.22
