Unable to use Kubeless triggers with Kubernetes 1.18

[kaivean]

**BUG REPORT **:

What happened:

create a function

kubeless function deploy hello --runtime nodejs12   --env REQ_MB_LIMIT=50     --handler main.hello     --from-file main.js

call and get a right response

kubeless function call hello --data 'Hello world111!'

create a http-trigger . But it can't create a ingress service
kubeless trigger http create hello-http --function-name hello

View logs By kubectl logs kubeless-controller-manager-54cc78b665-ksqkz -n kubeless -c http-trigger-controller the following log from the http-trigger-controller container.

time="2020-04-25T08:56:18Z" level=info msg="Running Kubeless HTTP trigger controller version: v1.0.1"
time="2020-04-25T08:56:18Z" level=info msg="Starting HTTP Trigger controller" controller=http-trigger-controller
time="2020-04-25T08:56:18Z" level=info msg="HTTP Trigger controller caches are synced and ready" controller=http-trigger-controller
time="2020-04-25T08:56:18Z" level=info msg="HTTP Trigger controller synced and ready" controller=http-trigger-controller
time="2020-04-25T10:49:23Z" level=info msg="Successfully processed update to function object hello Namespace: default" controller=http-trigger-controller
time="2020-04-25T10:49:23Z" level=info msg="Successfully processed update to function object hello Namespace: default" controller=http-trigger-controller
time="2020-04-25T10:50:26Z" level=info msg="Processing update to HTTPTrigger: default/hello-http" controller=http-trigger-controller
time="2020-04-25T10:50:26Z" level=info msg="Processing update to HTTPTrigger: default/hello-http" controller=http-trigger-controller
time="2020-04-25T10:50:26Z" level=info msg="Adding ingress resource for http trigger Obj: default/hello-http " controller=http-trigger-controller
time="2020-04-25T10:50:26Z" level=error msg="Failed to create ingress rule hello-http corresponding to http trigger Obj: default/hello-http due to: Unable to find the function internal service: &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:hello,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/services/hello,UID:7dbdbcc0-69eb-4146-be0e-f6044fdbf764,ResourceVersion:43169,Generation:0,CreationTimestamp:2020-04-25 10:49:23 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{created-by: kubeless,function: hello,},Annotations:map[string]string{},OwnerReferences:[{kubeless.io/v1beta1 Function hello f752134f-9f6b-44b6-9b0f-6ed7a638522f <nil> <nil>}],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[],Selector:map[string]string{},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},}: " controller=http-trigger-controller
time="2020-04-25T10:50:26Z" level=info msg="Processed update to HTTPTrigger: default/hello-http" controller=http-trigger-controller

What you expected to happen:

create a ingress service and be accessible by ingress

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:50:46Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

• Kubeless version (use kubeless version):

Kubeless version: v1.0.6-dirty

• Cloud provider or physical cluster:

aliyun centos 7.5 . ECS server

[kaivean]

I reproduced the isssue in two machines. thank for your reply.

[kaivean]

At the same time, I found it can create a cronjob trigger , but can't update the cronjob trigger

create a cronjob trigger

 kubeless trigger cronjob create   --function hello hellocron2 --schedule "0 * * * *"

update the cronjob trigger

kubeless trigger cronjob update   --function hello hellocron2 --schedule "*/1 * * * *"

the following log from kubectl logs kubeless-controller-manager-54cc78b665-ksqkz -n kubeless -c cronjob-trigger-controller

time="2020-04-25T11:40:38Z" level=info msg="Processing update to CronJob Trigger: default/hellocron2" controller=cronjob-trigger-controller
time="2020-04-25T11:40:38Z" level=error msg="Error processing default/hellocron2 (giving up): v1beta1.CronJob: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 753 ...:{},\"k:{\\\"... at {\"kind\":\"CronJob\",\"apiVersion\":\"batch/v1beta1\",\"metadata\":{\"name\":\"trigger-hello\",\"namespace\":\"default\",\"selfLink\":\"/apis/batch/v1beta1/namespaces/default/cronjobs/trigger-hello\",\"uid\":\"d6062765-c132-4dbb-bf8d-d3c568898de5\",\"resourceVersion\":\"50288\",\"creationTimestamp\":\"2020-04-25T11:40:04Z\",\"labels\":{\"created-by\":\"kubeless\",\"function\":\"hello\"},\"ownerReferences\":[{\"apiVersion\":\"kubeless.io/v1beta1\",\"kind\":\"Trigger\",\"name\":\"hellocron2\",\"uid\":\"e544ce08-e43e-4f14-8122-e83c350d6196\"}],\"managedFields\":[{\"manager\":\"cronjob-controller\",\"operation\":\"Update\",\"apiVersion\":\"batch/v1beta1\",\"time\":\"2020-04-25T11:40:04Z\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:labels\":{\".\":{},\"f:created-by\":{},\"f:function\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"e544ce08-e43e-4f14-8122-e83c350d6196\\\"}\":{\".\":{},\"f:apiVersion\":{},\"f:kind\":{},\"f:name\":{},\"f:uid\":{}}}},\"f:spec\":{\"f:concurrencyPolicy\":{},\"f:failedJobsHistoryLimit\":{},\"f:jobTemplate\":{\"f:spec\":{\"f:activeDeadlineSeconds\":{},\"f:template\":{\"f:spec\":{\"f:containers\":{\"k:{\\\"name\\\":\\\"trigger\\\"}\":{\".\":{},\"f:args\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:name\":{},\"f:resources\":{\".\":{},\"f:limits\":{\".\":{},\"f:cpu\":{},\"f:memory\":{}},\"f:requests\":{\".\":{},\"f:cpu\":{},\"f:memory\":{}}},\"f:terminationMessagePath\":{},\"f:terminationMessagePolicy\":{}}},\"f:dnsPolicy\":{},\"f:restartPolicy\":{},\"f:schedulerName\":{},\"f:securityContext\":{},\"f:terminationGracePeriodSeconds\":{}}}}},\"f:schedule\":{},\"f:successfulJobsHistoryLimit\":{},\"f:suspend\":{}}}}]},\"spec\":{\"schedule\":\"0 * * * *\",\"concurrencyPolicy\":\"Allow\",\"suspend\":false,\"jobTemplate\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"activeDeadlineSeconds\":180,\"template\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":[{\"name\":\"trigger\",\"image\":\"kubeless/unzip@sha256:4863100364496255de9bf8722a220dad7143ab277ac72435876eb8c93936e9d7\",\"args\":[\"curl\",\"-Lv\",\" -H \\\"event-id: FNYWXF6pic-4HZE\\\" -H \\\"event-time: 2020-04-25 11:40:04.501530306 +0000 UTC\\\" -H \\\"event-type: application/json\\\" -H \\\"event-namespace: cronjobtrigger.kubeless.io\\\"\",\"http://hello.default.svc.cluster.local:8080\"],\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"64Mi\"},\"requests\":{\"cpu\":\"10m\",\"memory\":\"16Mi\"}},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"IfNotPresent\"}],\"restartPolicy\":\"Never\",\"terminationGracePeriodSeconds\":30,\"dnsPolicy\":\"ClusterFirst\",\"securityContext\":{},\"schedulerName\":\"default-scheduler\"}}}},\"successfulJobsHistoryLimit\":3,\"failedJobsHistoryLimit\":1},\"status\":{}}" controller=cronjob-trigger-controller
E0425 11:40:38.754893       1 cronjob_trigger_controller.go:186] v1beta1.CronJob: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 753 ...:{},"k:{\"... at {"kind":"CronJob","apiVersion":"batch/v1beta1","metadata":{"name":"trigger-hello","namespace":"default","selfLink":"/apis/batch/v1beta1/namespaces/default/cronjobs/trigger-hello","uid":"d6062765-c132-4dbb-bf8d-d3c568898de5","resourceVersion":"50288","creationTimestamp":"2020-04-25T11:40:04Z","labels":{"created-by":"kubeless","function":"hello"},"ownerReferences":[{"apiVersion":"kubeless.io/v1beta1","kind":"Trigger","name":"hellocron2","uid":"e544ce08-e43e-4f14-8122-e83c350d6196"}],"managedFields":[{"manager":"cronjob-controller","operation":"Update","apiVersion":"batch/v1beta1","time":"2020-04-25T11:40:04Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:created-by":{},"f:function":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"e544ce08-e43e-4f14-8122-e83c350d6196\"}":{".":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:concurrencyPolicy":{},"f:failedJobsHistoryLimit":{},"f:jobTemplate":{"f:spec":{"f:activeDeadlineSeconds":{},"f:template":{"f:spec":{"f:containers":{"k:{\"name\":\"trigger\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}}},"f:schedule":{},"f:successfulJobsHistoryLimit":{},"f:suspend":{}}}}]},"spec":{"schedule":"0 * * * *","concurrencyPolicy":"Allow","suspend":false,"jobTemplate":{"metadata":{"creationTimestamp":null},"spec":{"activeDeadlineSeconds":180,"template":{"metadata":{"creationTimestamp":null},"spec":{"containers":[{"name":"trigger","image":"kubeless/unzip@sha256:4863100364496255de9bf8722a220dad7143ab277ac72435876eb8c93936e9d7","args":["curl","-Lv"," -H \"event-id: FNYWXF6pic-4HZE\" -H \"event-time: 2020-04-25 11:40:04.501530306 +0000 UTC\" -H \"event-type: application/json\" -H \"event-namespace: cronjobtrigger.kubeless.io\"","http://hello.default.svc.cluster.local:8080"],"resources":{"limits":{"cpu":"100m","memory":"64Mi"},"requests":{"cpu":"10m","memory":"16Mi"}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","securityContext":{},"schedulerName":"default-scheduler"}}}},"successfulJobsHistoryLimit":3,"failedJobsHistoryLimit":1},"status":{}}

[andresmgot]

Hi @kaivean,

The first issue seems a problem within your cluster. It seems that the pod in the kubeless namespace cannot reach the service hello in the namespace default. Can you check your cluster configuration to see if there is something that may affect this?

I am not able to reproduce your second issue either:

▶  kubeless trigger cronjob create   --function get-java-11 get-java-schedule --schedule "0 * * * *"
INFO[0000] Cronjob trigger get-java-schedule created in namespace default successfully! 

▶ kubectl get cronjob trigger-get-java-11 
NAME                  SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
trigger-get-java-11   0 * * * *   False     0        <none>          7s

▶ kubeless trigger cronjob update   --function get-java-11 get-java-schedule --schedule "*/1 * * * *"
INFO[0000] Cronjob trigger get-java-schedule updated in namespace default successfully! 

▶ kubectl get cronjob trigger-get-java-11                                                          
NAME                  SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
trigger-get-java-11   */1 * * * *   False     0        <none>          26s

Maybe Kubeless is running with limited permissions in your cluster?

[kaivean]

@andresmgot thanks for your reply.

limited permissions? please tell me about more details. I am new in k8s.

I try to create a function in namespace kubeless, but the issue still exists.

Now i downgrade to k8s 1.17, it works. so is it not compatible with k8s 1.18 ?

More infomation:

• centos: 7.5
• kernel: 4.14.0
• installing k8s by yum

[andresmgot]

Now i downgrade to k8s 1.17, it works. so is it not compatible with k8s 1.18 ?

mmh, that could be the reason, I haven't tested k8s 1.18 yet. Maybe it comes with some defaults or the k8s distribution installed with yum configures it in a way that makes it impossible to discover services (which would be weird). Can you check if the problem persists in a different distribution like Minikube?

[kaivean]

oh, sorry, I've never been used Minikube. I alse tried to install openfaas and openwhisk on k8s 1.18 in the same host, they work correctly.

[ricardorm13]

Now i downgrade to k8s 1.17, it works. so is it not compatible with k8s 1.18 ?

mmh, that could be the reason, I haven't tested k8s 1.18 yet. Maybe it comes with some defaults or the k8s distribution installed with yum configures it in a way that makes it impossible to discover services (which would be weird). Can you check if the problem persists in a different distribution like Minikube?

HI,

I'm getting the same issue running in a cluster with Kubernetes v1.18.2. I've installed following the k8s docs:

sudo apt-get update && sudo apt-get install -y apt-transport-https curl curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list deb https://apt.kubernetes.io/ kubernetes-xenial main EOF sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl

Have you found some solution?

[andresmgot]

Unfortunately no, it seems that there is some breaking change in Kubernetes 1.18 that prevents our service discovery to work as expected. I haven't had the time to investigate this yet, if someone can expend some time investigating the issue it would be highly appreciated.

[weiwei02]

I had the same issue.the logs as follow: time="2020-05-28T12:06:29Z" level=info msg="Adding ingress resource for http trigger Obj: default/get-python " controller=http-trigger-controller time="2020-05-28T12:06:29Z" level=error msg="Failed to create ingress rule get-python corresponding to http trigger Obj: default/get-python due to: Unable to find the function internal service: &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:get-python,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/services/get-python,UID:4e01f283-3380-451b-9505-6bdf22f4ba37,ResourceVersion:2509683,Generation:0,CreationTimestamp:2020-05-28 11:55:54 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{created-by: kubeless,function: get-python,},Annotations:map[string]string{},OwnerReferences:[{kubeless.io/v1beta1 Function get-python 7869cac7-198f-48bf-b9a6-cd2e0f56e8cf <nil> <nil>}],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[],Selector:map[string]string{},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},}: " controller=http-trigger-controller

the kubernetes information : Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

[dinurp]

I have the same issue.

Kubeless version: v1.0.7-dirty

Kubernetes version: Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

minikube version: v1.11.0 commit: 57e2f55f47effe9ce396cea42a1e0eb4f611ebbd

time="2020-06-30T18:38:25Z" level=error msg="Failed to create ingress rule hello corresponding to http trigger Obj: default/hello due to: Unable to find the function internal service: &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:hello,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/services/hello,UID:d1d717de-0ab5-4d52-943b-bb9adf033e07,ResourceVersion:20642,Generation:0,CreationTimestamp:2020-06-30 18:31:32 +0000 UTC,DeletionTimestamp:,DeletionGracePeriodSeconds:nil,Labels:map[string]string{created-by: kubeless,function: hello,},Annotations:map[string]string{},OwnerReferences:[{kubeless.io/v1beta1 Function hello 00c46b8e-ada9-4b48-b294-bb634df6b12c }],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[],Selector:map[string]string{},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},}: " controller=http-trigger-controller

[simodev25]

hello kaivean
I have the same problem with Kubernetes (1.18) at digitalocean I created another Kubernetes Clusters (1.17) it works. I think version 1.18 is not fully compatible.

[vinayagg]

I am also not able to get it to work in k8s 1.18, here are my logs

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

$ kubeless version
Kubeless version: 0.0.20 (bc472b73)

$ kubectl create ns kubeless
namespace/kubeless created

$ kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-v1.0.7.yaml
clusterrole.rbac.authorization.k8s.io/kubeless-controller-deployer created
clusterrolebinding.rbac.authorization.k8s.io/kubeless-controller-deployer created
customresourcedefinition.apiextensions.k8s.io/functions.kubeless.io created
customresourcedefinition.apiextensions.k8s.io/httptriggers.kubeless.io created
customresourcedefinition.apiextensions.k8s.io/cronjobtriggers.kubeless.io created
configmap/kubeless-config created
deployment.apps/kubeless-controller-manager created
serviceaccount/controller-acct created

$ kubectl -n kubeless get pods
NAME                                          READY   STATUS    RESTARTS   AGE
kubeless-controller-manager-59d484f4d-29ndx   3/3     Running   0          15s

$ kubectl -n kubeless logs -f kubeless-controller-manager-59d484f4d-29ndx -c kubeless-function-controller
time="2020-08-10T00:52:21Z" level=info msg="Running Kubeless controller manager version: v1.0.7"
time="2020-08-10T00:52:22Z" level=info msg="Starting Function controller" pkg=function-controller
time="2020-08-10T00:52:22Z" level=info msg="Function controller synced and ready" pkg=function-controller

$ kubeless function list
FATA[0000] the server could not find the requested resource (get functions.k8s.io)

[vinayagg]

I switched to docker-desktop 1.16 and faced same issue. here is my bash log

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

$ kubeless version
Kubeless version: 0.0.20 (bc472b73)

$ kubectl create ns kubeless
namespace/kubeless created

$ kubectl create -f https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml

configmap/kubeless-config created
deployment.apps/kubeless-controller-manager created
serviceaccount/controller-acct created
Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml": customresourcedefinitions.apiextensions.k8s.io "functions.kubeless.io" already exists
Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml": customresourcedefinitions.apiextensions.k8s.io "httptriggers.kubeless.io" already exists
Error from server (AlreadyExists): error when creating "https://github.com/kubeless/kubeless/releases/download/v1.0.7/kubeless-non-rbac-v1.0.7.yaml": customresourcedefinitions.apiextensions.k8s.io "cronjobtriggers.kubeless.io" already exists

$ kubectl -n kubeless get pods
NAME                                           READY   STATUS    RESTARTS   AGE
kubeless-controller-manager-7ccdccb78b-f6zzb   3/3     Running   0          16s

$ kubectl -n kubeless logs -f kubeless-controller-manager-7ccdccb78b-f6zzb -c kubeless-function-controller
time="2020-08-10T01:09:16Z" level=info msg="Running Kubeless controller manager version: v1.0.7"
time="2020-08-10T01:09:16Z" level=info msg="Starting Function controller" pkg=function-controller
time="2020-08-10T01:09:16Z" level=info msg="Function controller synced and ready" pkg=function-controller


$ kubeless function list
FATA[0000] the server could not find the requested resource (get functions.k8s.io)

[andresmgot]

@vinayagg that's a very old version of Kubeless. Please try with a recent version, it should work fine for Kubernetes 1.16.

[vinayagg]

@andresmgot Can you guide little further. As per releases page https://github.com/kubeless/kubeless/releases version 1.0.7 is latest version released on Jun 30.

If this is not latest version, where can I find latest version?

[andresmgot]

Sorry, I meant the CLI version, from your output your are using 0.0.20 rather than 1.0.7:

$ kubeless version
Kubeless version: 0.0.20 (bc472b73)

You need to download a newer version and put it in your PATH.

[vinayagg]

Thank you so much. That was the issue. Kubeless works now.

[sadoMasupilami]

hey, i hit the same problem with a rancher installed bare metal cluster using kubernetes v1.18.3

[sadoMasupilami]

i did a little testing. The service itself is working fine:

I am not very sure but i think the service is missing some annotations from the error mesage:

time="2020-08-22T09:52:33Z" level=error msg="Failed to create ingress rule hello corresponding to http trigger Obj: kubeless/hello due to: Unable to find the function internal service: &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:hello,GenerateName:,Namespace:kubeless,SelfLink:/api/v1/namespaces/kubeless/services/hello,UID:fbf704e4-261c-42ce-a134-f13742fead4f,ResourceVersion:37010626,Generation:0,CreationTimestamp:2020-08-22 09:35:39 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{created-by: kubeless,function: hello,},Annotations:map[string]string{OwnerReferences: kubeless.io/v1beta1 Function hello c6459e44-4461-4533-9372-279660ab960a,field.cattle.io/ipAddresses: null,field.cattle.io/targetDnsRecordIds: null,field.cattle.io/targetWorkloadIds: null,},OwnerReferences:[{kubeless.io/v1beta1 Function hello c6459e44-4461-4533-9372-279660ab960a <nil> <nil>}],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[],Selector:map[string]string{},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},}: " controller=http-trigger-controller

could it be that there is no annotatio that kubeless is searching? Annotations:map[string]string{OwnerReferences: kubeless.io/v1beta1 Function hello c6459e44-4461-4533-9372-279660ab960a,field.cattle.io/ipAddresses: null,field.cattle.io/targetDnsRecordIds: null,field.cattle.io/targetWorkloadIds: null,}

Here my service yaml output:

maybe someone could compare the service to a 1.17- cluster i hope this is a little useful analyzing the problem :-)

[LibraSkywalker]

same issue here. k8s version: 1.18 kubeless version: v1.0.7-dirty

$ kubectl logs -n kubeless -l kubeless=controller -c http-trigger-controller
time="2020-09-08T14:35:06Z" level=info msg="Successfully processed update to function object test Namespace: default" controller=http-trigger-controller
time="2020-09-08T14:35:06Z" level=info msg="Function test deleted. Removing associated http trigger" controller=http-trigger-controller
time="2020-09-08T14:35:06Z" level=info msg="Successfully processed update to function object test Namespace: default" controller=http-trigger-controller
time="2020-09-08T14:35:10Z" level=info msg="Successfully processed update to function object test Namespace: default" controller=http-trigger-controller
time="2020-09-08T14:35:10Z" level=info msg="Successfully processed update to function object test Namespace: default" controller=http-trigger-controller
time="2020-09-08T14:35:10Z" level=info msg="Processing update to HTTPTrigger: default/test" controller=http-trigger-controller
time="2020-09-08T14:35:10Z" level=info msg="Processing update to HTTPTrigger: default/test" controller=http-trigger-controller
time="2020-09-08T14:35:10Z" level=info msg="Adding ingress resource for http trigger Obj: default/test " controller=http-trigger-controller
time="2020-09-08T14:35:10Z" level=error msg="Failed to create ingress rule test corresponding to http trigger Obj: default/test due to: Unable to find the function internal service: &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:test,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/services/test,UID:3b5f3ecf-f715-4e80-81e6-e559c80d6cfe,ResourceVersion:4870450,Generation:0,CreationTimestamp:2020-09-08 14:35:10 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{created-by: kubeless,function: test,},Annotations:map[string]string{},OwnerReferences:[{kubeless.io/v1beta1 Function test 75f62353-9fed-48a3-af13-da3a07ff314f <nil> <nil>}],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[],Selector:map[string]string{},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},}: " controller=http-trigger-controller
time="2020-09-08T14:35:10Z" level=info msg="Processed update to HTTPTrigger: default/test" controller=http-trigger-controller

[rashkur]

Same here. Tried on a fresly bootstraped cluster.

Kubeless version: v1.0.7-dirty Kops: Version 1.18.0 Kubect: Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"archive", BuildDate:"2020-09-03T15:34:56Z", GoVersion:"go1.15.1", Compiler:"gc", Platform:"linux/amd64"}

kubectl logs kubeless-controller-manager-59d484f4d-97plz -n kubeless -c http-trigger-controller

...
time="2020-09-08T15:55:58Z" level=error msg="Failed to create ingress rule myfunction corresponding to http trigger Obj: default/myfunction due to: Unable to find the function internal service: &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:myfunction,GenerateName:,Namespace:default,SelfLink:/api/v1/namespaces/default/services/myfunction,UID:bd3ba5fd-077c-49c2-b57b-13f5aa5b3194,ResourceVersion:1665,Generation:0,CreationTimestamp:2020-09-08 15:41:04 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{created-by: kubeless,function: myfunction,},Annotations:map[string]string{},OwnerReferences:[{kubeless.io/v1beta1 Function myfunction 4ae11168-9945-4732-95ac-5f30355f0d83 <nil> <nil>}],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[],Selector:map[string]string{},ClusterIP:,Type:,ExternalIPs:[],SessionAffinity:,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},}: " controller=http-trigger-controller

[yalopov]

Can confirm it is working as it should using k8s v1.17.11

[asishrs]

Seeing the same issue on k8s v1.19 as well.

[hjanuschka]

first of all 🤚 and thx for the great project. any - ETA when this will, or if this will work - on 1.19? are there any workarounds, it pretty much makes it not really prod-ready for me, can i somehow help?

[hjanuschka]

it is definitly this part here: https://github.com/kubeless/http-trigger/blob/232119f28afb3cf0b97b05d040bcec42d3bac50a/pkg/utils/k8sutil.go#L227

however i am not able how to fix. testing the api thing with kubectl -n default get svc functionname / also with ruby & node sdk.

[hjanuschka]

see https://github.com/kubeless/http-trigger/pull/14 got it working for my 1.19 cluster

[andresmgot]

The kafka-trigger is now compatible with Kubernetes +1.18 (latest release: 1.1.0: https://github.com/kubeless/kafka-trigger/releases/tag/v1.1.0)

The http-trigger support is in the works, thanks @hjanuschka and @sepetrov!

[russbeye]

Hello, and great work with getting this narrowed down! @andresmgot, has there been an update on the http-trigger?

[andresmgot]

cc/ @sepetrov

I think that's still on the works, if someone wants to help, the PR is here: https://github.com/kubeless/http-trigger/pull/15

[jnsvd]

The kafka-trigger is now compatible with Kubernetes +1.18 (latest release: 1.1.0: https://github.com/kubeless/kafka-trigger/releases/tag/v1.1.0)

The http-trigger support is in the works, thanks @hjanuschka and @sepetrov!

I'm still getting this issue. I have a function running in the kubeless namespace, but the controller is unable to find the service.