gravity
gravity copied to clipboard
Published
20 hours ago •
gravitational
gravitational
[BUG] kubectl exec fails remotely
error: Internal error occurred: error executing command in container: Unauthorized
Gravity Version: 6.3.7 It is reported as a regression from 6.1.18.
No, sorry - haven't had a chance to look into this yet.
@guykanyvision @snirkatriel are you having this issue with 6.3.x or any other version?
@snirkatriel Not yet. I'm sorry I've not been able to look into this yet. I will do it this week.
@snirkatriel Tried to reproduce on a development build of 6.3.x to no avail. Can you share the kubectl logs when running kubectl --v=10 exec -ti ... ?
@snirkatriel Tried to reproduce on a development build of 6.3.x to no avail. Can you share the kubectl logs when running kubectl --v=10 exec -ti ... ?
@snirkatriel Tried to reproduce on a development build of 6.3.x to no avail. Can you share the kubectl logs when running kubectl --v=10 exec -ti ... ?
@snirkatriel Tried to reproduce on a development build of 6.3.x to no avail. Can you share the kubectl logs when running kubectl --v=10 exec -ti ... ?
This is the output @a-palchikov:
I0423 16:56:43.975763 27630 loader.go:375] Config loaded from file: /home/guyk/.kube/config
I0423 16:56:43.977709 27630 cached_discovery.go:114] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/servergroups.json
I0423 16:56:43.978150 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apiregistration.k8s.io/v1/serverresources.json
I0423 16:56:43.978188 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/certificates.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.978208 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apps/v1/serverresources.json
I0423 16:56:43.978245 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apiregistration.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.978250 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/extensions/v1beta1/serverresources.json
I0423 16:56:43.978265 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/authentication.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.978271 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apps/v1beta1/serverresources.json
I0423 16:56:43.978289 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/project.cattle.io/v3/serverresources.json
I0423 16:56:43.978286 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/autoscaling/v2beta1/serverresources.json
I0423 16:56:43.978334 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/autoscaling/v2beta2/serverresources.json
I0423 16:56:43.978335 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/v1/serverresources.json
I0423 16:56:43.978352 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/batch/v1beta1/serverresources.json
I0423 16:56:43.978349 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/authorization.k8s.io/v1/serverresources.json
I0423 16:56:43.978379 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/authorization.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.978385 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/batch/v2alpha1/serverresources.json
I0423 16:56:43.978402 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/autoscaling/v1/serverresources.json
I0423 16:56:43.978404 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apps/v1beta2/serverresources.json
I0423 16:56:43.978419 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/batch/v1/serverresources.json
I0423 16:56:43.978417 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/events.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.978480 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/scheduling.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.978537 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/coordination.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.978610 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/admissionregistration.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.978482 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/node.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.978509 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/argoproj.io/v1alpha1/serverresources.json
I0423 16:56:43.978521 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/monitoring.coreos.com/v1/serverresources.json
I0423 16:56:43.979098 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/rbac.authorization.k8s.io/v1/serverresources.json
I0423 16:56:43.978526 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/coordination.k8s.io/v1/serverresources.json
I0423 16:56:43.978547 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/authentication.k8s.io/v1/serverresources.json
I0423 16:56:43.978552 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/networking.k8s.io/v1/serverresources.json
I0423 16:56:43.978548 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/discovery.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.978575 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/changeset.gravitational.io/v1/serverresources.json
I0423 16:56:43.978614 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apiextensions.k8s.io/v1/serverresources.json
I0423 16:56:43.978614 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/networking.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.978789 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/kafka.strimzi.io/v1beta1/serverresources.json
I0423 16:56:43.978896 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/policy/v1beta1/serverresources.json
I0423 16:56:43.978908 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/acme.cert-manager.io/v1alpha2/serverresources.json
I0423 16:56:43.978913 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/admissionregistration.k8s.io/v1/serverresources.json
I0423 16:56:43.978922 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/cert-manager.io/v1alpha2/serverresources.json
I0423 16:56:43.979193 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/rbac.authorization.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.978918 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/scheduling.k8s.io/v1/serverresources.json
I0423 16:56:43.978490 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/storage.k8s.io/v1/serverresources.json
I0423 16:56:43.979005 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/storage.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.979019 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apiextensions.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.979250 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/kafka.strimzi.io/v1alpha1/serverresources.json
I0423 16:56:43.979805 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/management.cattle.io/v3/serverresources.json
I0423 16:56:43.980055 27630 cached_discovery.go:114] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/servergroups.json
I0423 16:56:43.980188 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/project.cattle.io/v3/serverresources.json
I0423 16:56:43.980192 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/authorization.k8s.io/v1/serverresources.json
I0423 16:56:43.980193 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/policy/v1beta1/serverresources.json
I0423 16:56:43.980197 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/events.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980226 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apps/v1beta2/serverresources.json
I0423 16:56:43.980234 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apiregistration.k8s.io/v1/serverresources.json
I0423 16:56:43.980240 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apps/v1/serverresources.json
I0423 16:56:43.980249 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/authentication.k8s.io/v1/serverresources.json
I0423 16:56:43.980255 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/authorization.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980278 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/batch/v1beta1/serverresources.json
I0423 16:56:43.980285 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/rbac.authorization.k8s.io/v1/serverresources.json
I0423 16:56:43.980294 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apps/v1beta1/serverresources.json
I0423 16:56:43.980305 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apiregistration.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980310 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/authentication.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980315 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/batch/v2alpha1/serverresources.json
I0423 16:56:43.980337 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/autoscaling/v1/serverresources.json
I0423 16:56:43.980357 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/networking.k8s.io/v1/serverresources.json
I0423 16:56:43.980369 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/rbac.authorization.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980378 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/networking.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980401 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/certificates.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980418 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/autoscaling/v2beta2/serverresources.json
I0423 16:56:43.980432 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/autoscaling/v2beta1/serverresources.json
I0423 16:56:43.980433 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/extensions/v1beta1/serverresources.json
I0423 16:56:43.980444 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/storage.k8s.io/v1/serverresources.json
I0423 16:56:43.980446 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/batch/v1/serverresources.json
I0423 16:56:43.980459 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/node.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980478 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/v1/serverresources.json
I0423 16:56:43.980493 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/acme.cert-manager.io/v1alpha2/serverresources.json
I0423 16:56:43.980502 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/changeset.gravitational.io/v1/serverresources.json
I0423 16:56:43.980538 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/storage.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980543 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/kafka.strimzi.io/v1beta1/serverresources.json
I0423 16:56:43.980544 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/monitoring.coreos.com/v1/serverresources.json
I0423 16:56:43.980543 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/cert-manager.io/v1alpha2/serverresources.json
I0423 16:56:43.980560 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/discovery.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980588 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/kafka.strimzi.io/v1alpha1/serverresources.json
I0423 16:56:43.980602 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/scheduling.k8s.io/v1/serverresources.json
I0423 16:56:43.980604 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/admissionregistration.k8s.io/v1/serverresources.json
I0423 16:56:43.980610 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/argoproj.io/v1alpha1/serverresources.json
I0423 16:56:43.980617 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apiextensions.k8s.io/v1/serverresources.json
I0423 16:56:43.980616 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/coordination.k8s.io/v1/serverresources.json
I0423 16:56:43.980637 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/apiextensions.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980652 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/coordination.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980656 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/scheduling.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980670 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/admissionregistration.k8s.io/v1beta1/serverresources.json
I0423 16:56:43.980799 27630 cached_discovery.go:71] returning cached discovery info from /home/guyk/.kube/cache/discovery/35.197.27.42_3026/management.cattle.io/v3/serverresources.json
I0423 16:56:43.982002 27630 round_trippers.go:423] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.17.4 (linux/amd64) kubernetes/8d8aa39" 'https://35.197.27.42:3026/api/v1/namespaces/default/pods/consul-server-0'
I0423 16:56:44.936675 27630 round_trippers.go:443] GET https://35.197.27.42:3026/api/v1/namespaces/default/pods/consul-server-0 200 OK in 954 milliseconds
I0423 16:56:44.936726 27630 round_trippers.go:449] Response Headers:
I0423 16:56:44.936757 27630 round_trippers.go:452] Cache-Control: no-cache, no-store, must-revalidate
I0423 16:56:44.936794 27630 round_trippers.go:452] Content-Type: application/json
I0423 16:56:44.936816 27630 round_trippers.go:452] Date: Thu, 23 Apr 2020 13:56:44 GMT
I0423 16:56:44.936843 27630 round_trippers.go:452] Expires: 0
I0423 16:56:44.936862 27630 round_trippers.go:452] Pragma: no-cache
I0423 16:56:44.937150 27630 request.go:1017] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"consul-server-0","generateName":"consul-server-","namespace":"default","uid":"8206b963-698d-475a-afef-8dcdd379fc8c","resourceVersion":"15444","creationTimestamp":"2020-04-23T13:43:45Z","labels":{"app":"consul","chart":"consul-helm","component":"server","controller-revision-hash":"consul-server-686f4f9dbf","hasDNS":"true","release":"core-data","statefulset.kubernetes.io/pod-name":"consul-server-0"},"annotations":{"consul.hashicorp.com/connect-inject":"false"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"StatefulSet","name":"consul-server","uid":"512da20d-f416-4bd1-a52e-496ae855d57c","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"data-default","persistentVolumeClaim":{"claimName":"data-default-consul-server-0"}},{"name":"config","configMap":{"name":"consul-server-config","defaultMode":420}},{"name":"consul-server-token-h95bj","secret":{"secretName":"consul-server-token-h95bj","defaultMode":420}}],"containers":[{"name":"consul","image":"consul:1.4.4","command":["/bin/sh","-ec","CONSUL_FULLNAME=\"consul\"\n\nexec /bin/consul agent \\\n -advertise=\"${POD_IP}\" \\\n -bind=0.0.0.0 \\\n -bootstrap-expect=1 \\\n -client=0.0.0.0 \\\n -config-dir=/consul/config \\\n -datacenter=dc1 \\\n -data-dir=/consul/data \\\n -domain=consul \\\n -hcl=\"connect { enabled = true }\" \\\n -ui \\\n -retry-join=${CONSUL_FULLNAME}-server-0.${CONSUL_FULLNAME}-server.${NAMESPACE}.svc \\\n -server\n"],"ports":[{"name":"http","containerPort":8500,"protocol":"TCP"},{"name":"serflan","containerPort":8301,"protocol":"TCP"},{"name":"serfwan","containerPort":8302,"protocol":"TCP"},{"name":"server","containerPort":8300,"protocol":"TCP"},{"name":"dns-tcp","containerPort":8600,"protocol":"TCP"},{"name":"dns-udp","containerPort":8600,"protocol":"UDP"}],"env":[{"name":"POD_IP","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"status.podIP"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"volumeMounts":[{"name":"data-default","mountPath":"/consul/data"},{"name":"config","mountPath":"/consul/config"},{"name":"consul-server-token-h95bj","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"readinessProbe":{"exec":{"command":["/bin/sh","-ec","curl http://127.0.0.1:8500/v1/status/leader 2\u003e/dev/null | \\\ngrep -E '\".+\"'\n"]},"initialDelaySeconds":5,"timeoutSeconds":5,"periodSeconds":3,"successThreshold":1,"failureThreshold":2},"lifecycle":{"preStop":{"exec":{"command":["/bin/sh","-c","consul leave"]}}},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":10,"dnsPolicy":"ClusterFirst","serviceAccountName":"consul-server","serviceAccount":"consul-server","nodeName":"10.138.0.22","securityContext":{"fsGroup":1000},"hostname":"consul-server-0","subdomain":"consul-server","affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"app":"consul","component":"server","release":"core-data"}},"topologyKey":"kubernetes.io/hostname"}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-23T13:43:45Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-23T13:44:00Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-23T13:44:00Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-23T13:43:45Z"}],"hostIP":"10.138.0.22","podIP":"10.244.83.72","podIPs":[{"ip":"10.244.83.72"}],"startTime":"2020-04-23T13:43:45Z","containerStatuses":[{"name":"consul","state":{"running":{"startedAt":"2020-04-23T13:43:51Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"consul:1.4.4","imageID":"docker-pullable://consul@sha256:61ad29a7b0eac9c0a54a2719fd9e34e51b6dca987da94d7562fe1b57ec604c2a","containerID":"docker://fdda50ae741e22819022ae77fc0771f807efe93dfdffe68072f4e8b48f84bc50","started":true}],"qosClass":"BestEffort"}}
I0423 16:56:44.961398 27630 round_trippers.go:423] curl -k -v -XPOST -H "User-Agent: kubectl/v1.17.4 (linux/amd64) kubernetes/8d8aa39" -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" 'https://35.197.27.42:3026/api/v1/namespaces/default/pods/consul-server-0/exec?command=%2Fbin%2Fsh&container=consul&stdin=true&stdout=true&tty=true'
I0423 16:56:45.865215 27630 round_trippers.go:443] POST https://35.197.27.42:3026/api/v1/namespaces/default/pods/consul-server-0/exec?command=%2Fbin%2Fsh&container=consul&stdin=true&stdout=true&tty=true 101 Switching Protocols in 903 milliseconds
I0423 16:56:45.865262 27630 round_trippers.go:449] Response Headers:
I0423 16:56:45.865282 27630 round_trippers.go:452] Pragma: no-cache
I0423 16:56:45.865298 27630 round_trippers.go:452] Upgrade: SPDY/3.1
I0423 16:56:45.865312 27630 round_trippers.go:452] X-Stream-Protocol-Version: v4.channel.k8s.io
I0423 16:56:45.865330 27630 round_trippers.go:452] Date: Thu, 23 Apr 2020 13:56:45 GMT
I0423 16:56:45.865344 27630 round_trippers.go:452] Cache-Control: no-cache, no-store, must-revalidate
I0423 16:56:45.865359 27630 round_trippers.go:452] Connection: Upgrade
I0423 16:56:45.865374 27630 round_trippers.go:452] Expires: 0
F0423 16:56:46.766285 27630 helpers.go:114] error: Internal error occurred: error executing command in container: Unauthorized
@guykanyvision: sorry again for delay.
It is not quite clear even with the debug output from kubectl exec. Can you additionally share the details regarding what roles does the user you use to tsh login with has assigned and whether you are allowed to exec into that pod:
$ kubectl auth can-i exec pods/consul-server-0