consul-registration-hook icon indicating copy to clipboard operation
consul-registration-hook copied to clipboard

Published 20 hours ago • verified publisher icon allegro

Detail Document Needed to Setup consul-registration-hook

Open ashishchalak opened this issue 4 years ago • 13 comments

Is there any detailed documentation available to setup consul-registration-hook with kubernetes. My requirement is to register PODS and its IP address with Consul.

ashishchalak avatar Jul 14 '20 12:07 ashishchalak

[myhost:/home/users/ashishrajshek/consul-registration-hook/cmd/consul-registration-hook]$ export KUBERNETES_SERVICE_HOST=10.73.28.78 [myhost:/home/users/ashishrajshek/consul-registration-hook/cmd/consul-registration-hook]$ export KUBERNETES_SERVICE_PORT=6443 [myhost:/home/users/ashishrajshek/consul-registration-hook/cmd/consul-registration-hook]$ ./consul-registration-hook register k8s 2020/07/14 05:43:55 consul-registration-hook (version: ) 2020/07/14 05:43:55 Registering services using data from Kubernetes API 2020/07/14 05:43:55 error getting services to register: unable create K8S API client: couldn't initialize client: open /var/run/secrets/kubernetes.io/serviceaccount/namespace: no such file or directory

ashishchalak avatar Jul 14 '20 12:07 ashishchalak

Are You running this command from inside of the running POD? In /var/run/secrets/kubernetes.io/serviceaccount/ should be default serviceaccount data as K8S documentation states:

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod

ojagodzinski avatar Jul 14 '20 13:07 ojagodzinski

Hi Oskar,

Thanks for your quick response. Let me check via POD and get back.

ashishchalak avatar Jul 14 '20 15:07 ashishchalak

What is the value I should be passing in GS_URL, KUBERNETES_POD_NAME, KUBERNETES_POD_NAMESPACE and HOST_IP in below yaml file. I have a consul running on http://10.73.28.78:8500

apiVersion: v1 kind: Pod metadata: name: pod-with-consul-hook labels: consul: service-name spec: initContainers:

  • name: hook-init-container image: google/cloud-sdk:alpine imagePullPolicy: Always command: ["/bin/sh"] args: ["-c", "gsutil cat ${GS_URL} | tar -C /hooks -zxvf -"] env:
    • name: GS_URL value: 'http://10.73.28.78:8500' volumeMounts:
    • name: hooks mountPath: /hooks containers:
  • name: service-with-consul-hook-container image: python:2 command: ["python", "-m", "SimpleHTTPServer", "8080"] env:
    • name: KUBERNETES_POD_NAME value: 'ashishpodname'
    • name: KUBERNETES_POD_NAMESPACE value: 'default'
    • name: HOST_IP value: 'http://10.73.28.78:8500'
    • name: CONSUL_HTTP_ADDR value: "$(HOST_IP):8500" ports:
    • containerPort: 8080 volumeMounts:
    • name: hooks mountPath: /hooks lifecycle: postStart: exec: command: ["/bin/sh", "-c", "/hooks/consul-registration-hook register k8s"] preStop: exec: command: ["/bin/sh", "-c", "/hooks/consul-registration-hook deregister k8s"] volumes:
  • name: hooks emptyDir: {}

ashishchalak avatar Jul 14 '20 16:07 ashishchalak

error:

2020/07/14 05:43:55 error getting services to register: unable create K8S API client: couldn't initialize client: open /var/run/secrets/kubernetes.io/serviceaccount/namespace: no such file or directory

Has nothing to do with environment variables and their values. Problem lies in the lack of a serviceaccount directory inside POD. There (in POD where hook is executed) should be directory as below:

~$kubectl run my-shell --rm -i --tty --image ubuntu:latest -- bash
root@my-shell-67f75f7478-kds4r:/# ls -al /var/run/secrets/kubernetes.io/serviceaccount/
drwxr-xr-x 2 root root  100 Jul 14 00:00 ..2020_07_14_00_00_00
lrwxrwxrwx 1 root root   31 Jul 14 00:00 ..data -> ..2020_07_14_00_00_00
lrwxrwxrwx 1 root root   13 Jul 14 00:00 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root   16 Jul 14 00:00 namespace -> ..data/namespace
lrwxrwxrwx 1 root root   12 Jul 14 00:00 token -> ..data/token

ojagodzinski avatar Jul 14 '20 17:07 ojagodzinski

I also have access to Service Account directory

[myhost:/home/users/ashishrajshek]$ kubectl run my-shell --rm -i --tty --image ubuntu:latest -- bash If you don't see a command prompt, try pressing enter. root@my-shell:/# ls -al /var/run/secrets/kubernetes.io/serviceaccount/ total 0 drwxrwxrwt 3 root root 140 Jul 14 17:20 . drwxr-xr-x 3 root root 28 Jul 14 17:21 .. drwxr-xr-x 2 root root 100 Jul 14 17:20 ..2020_07_14_17_20_58.384039542 lrwxrwxrwx 1 root root 31 Jul 14 17:20 ..data -> ..2020_07_14_17_20_58.384039542 lrwxrwxrwx 1 root root 13 Jul 14 17:20 ca.crt -> ..data/ca.crt lrwxrwxrwx 1 root root 16 Jul 14 17:20 namespace -> ..data/namespace lrwxrwxrwx 1 root root 12 Jul 14 17:20 token -> ..data/token root@my-shell:/#

ashishchalak avatar Jul 14 '20 17:07 ashishchalak

Getting this exception when trying to register via POD.

$kubectl describe pod pod-with-consul-hook

Events:
Type Reason Age From Message

Normal Scheduled default-scheduler Successfully assigned default/pod-with-consul-hook to myhost
Normal Pulled 12s kubelet, myhost Container image "python:2" already present on machine
Normal Created 12s kubelet, myhost Created container service-with-consul-hook-container
Normal Started 12s kubelet, myhost Started container service-with-consul-hook-container
Warning FailedPostStartHook 1s kubelet, myhost Exec lifecycle hook ([/bin/sh -c /hooks/hooks/consul-registration-hook register k8s]) for Container "service-with-consul-hook-container" in Pod "pod-with-consul-hook_default(51624215-6b8a-4d64-9790-cf0d0902fd1a)" failed - error: command '/bin/sh -c /hooks/hooks/consul-registration- hook register k8s' exited with 1: 2020/07/14 18:00:56 consul-registration-hook (version: )
2020/07/14 18:00:56 Registering services using data from Kubernetes API
2020/07/14 18:01:05 unable to get pod data from API: unable to get pod data from API: kubernetes api: Failure 403 pods "pod-with-consul-hook" is forbidden: User "system:serviceaccount:default:default" cannot get resource "pods" in API group "" in the namespace "default"
2020/07/14 18:01:06 error getting services to register: unable to get pod data from API: could not get valid Pod data after 10s

ashishchalak avatar Jul 14 '20 18:07 ashishchalak 

2020/07/14 18:01:05 unable to get pod data from API: unable to get pod data from API: kubernetes api: Failure 403 pods "pod-with-consul-hook" is forbidden: User "system:serviceaccount:default:default" cannot get resource "pods" in API group "" in the namespace "default"

serviceaccount injected to POD must have broader permissions.

https://kubernetes.io/docs/reference/access-authn-authz/rbac/#service-account-permissions

ojagodzinski avatar Jul 15 '20 07:07 ojagodzinski

I elevated the permissions to admin role.

In POD Events i get the below exceptions:

Warning FailedPostStartHook 3m17s kubelet, myhost.com Exec lifecycle hook ([/bin/sh -c /hooks/hooks/consul-registration-hook register k8s]) for Container "service-with-consul-hook-container" in Pod "pod-with-consul-hook_default(624441b4-5fb8-432d-a72c-ab12a4015551)" failed - error: command '/bin/sh -c /hooks/hooks/consul-registration-hook register k8s' exited with 1: 2020/07/15 08:13:02 consul-registration-hook (version: )
2020/07/15 08:13:02 Registering services using data from Kubernetes API
2020/07/15 08:13:12 error getting services to register: unable to get pod data from API: could not get valid Pod data after 10s, message: "

When I try to run same command from inside the POD, Service is registered.

[myhost:/home/users/ashishrajshek/consul-registration-hook]$ kubectl exec -it pod-with-consul-hook sh kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.

/hooks/hooks/consul-registration-hook register k8s

2020/07/15 08:35:21 consul-registration-hook (version: ) 2020/07/15 08:35:21 Registering services using data from Kubernetes API 2020/07/15 08:35:21 Won't include failure domain data in registration: failure domain labels don't exist 2020/07/15 08:35:21 no port configuration (PORT_DEFINITIONS) 2020/07/15 08:35:21 Found 1 services to register 2020/07/15 08:35:21 Registering "service-name" service in Consul agent

/hooks/hooks/consul-registration-hook deregister k8s

2020/07/15 08:35:46 consul-registration-hook (version: ) 2020/07/15 08:35:46 Deregistering services using data from Kubernetes API 2020/07/15 08:35:46 Won't include failure domain data in registration: failure domain labels don't exist 2020/07/15 08:35:46 no port configuration (PORT_DEFINITIONS) 2020/07/15 08:35:46 Found 1 services to deregister 2020/07/15 08:35:46 Deregistering "10.244.0.212_8080" service in Consul agent

ashishchalak avatar Jul 15 '20 08:07 ashishchalak

When I try to run same command from inside the POD, Service is registered.

So everything works, hook should be executed from inside of the POD.

ojagodzinski avatar Jul 15 '20 08:07 ojagodzinski

Are logs stored anywhere inside POD while registering and de registering services. If yes which location?

ashishchalak avatar Jul 15 '20 11:07 ashishchalak

Hook prints logs to stdout/err.

ojagodzinski avatar Jul 15 '20 12:07 ojagodzinski

Any idea on this exception:

Warning FailedPostStartHook 40s (x3 over 66s) kubelet, myhost.com Exec lifecycle hook ([/bin/sh -c /hooks/hooks/consul-registration-hook register k8s]) for Container "cidtestserver" in Pod "cidtestserver-2_default(a56d696a-e3df-4a07-8d86-65a36e38f284)" failed - error: command '/bin/sh -c /hooks/hooks/consul-registration-hook register k8s' exited with 126: /bin/sh: /hooks/hooks/consul-registration-hook: Permission denied , message: "/bin/sh: /hooks/hooks/consul-registration-hook: Permission denied\n"

ashishchalak avatar Jul 15 '20 13:07 ashishchalak