telepresence connect failure when using server versions older than 1.17.

[ManigandanMS]

I have remote kubernetes cluster and have done port forwarding to kube api port number 9443 on my local mac. I can execute kubectl commands which returns expected output. Have installed telepresence successfully but unable to connect. When i use telepresence connect i get below error

Connecting to traffic manager... Disconnecting...done telepresence: error: connector.Connect: failed to start traffic manager: failed to create MutatingWebhookConfiguration agent-injector-webhook.ambassador: MutatingWebhookConfiguration.admissionregistration.k8s.io "agent-injector-webhook" is invalid: webhooks[0].admissionReviewVersions: Invalid value: []string{"v1"}: none of the versions accepted by this server. accepted version(s) are v1beta1

Telepresence version: Client: v2.3.1 (api v3) Root Daemon: v2.3.1 (api v3) User Daemon: not running

[darinpp]

I have the same problem

Launching Telepresence Daemon v2.3.1 (api v3)
Connecting to traffic manager...
Disconnecting...done
Telepresence Daemon quitting...done
telepresence: error: connector.Connect: failed to start traffic manager: failed to create MutatingWebhookConfiguration agent-injector-webhook.ambassador: MutatingWebhookConfiguration.admissionregistration.k8s.io "agent-injector-webhook" is invalid: webhooks[0].admissionReviewVersions: Invalid value: []string{"v1"}: must include at least one of v1beta1

Here is the version

kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15", GitCommit:"2adc8d7091e89b6e3ca8d048140618ec89b39369", GitTreeState:"clean", BuildDate:"2020-09-02T11:40:00Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.15+k3s1", GitCommit:"790b740f0108f1fc1d17d77cb69f882bddf6dae9", GitTreeState:"clean", BuildDate:"2020-09-04T03:09:08Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

I'm using k3d/k3s

k3d version
k3d version v4.2.0
k3s version v1.20.2-k3s1 (default)

[darinpp]

And a log file if that helps

cat connector-20210616T082811.log
2021/06/15 19:19:07.7824 info     : No config found. Using default (from pkg/client/config.go:373)
2021/06/15 19:19:07.7827 info    connector : --- (from pkg/client/connector/service.go:571)
2021/06/15 19:19:07.7827 info    connector : Telepresence Connector v2.3.1 (api v3) starting... (from pkg/client/connector/service.go:572)
2021/06/15 19:19:07.7828 info    connector : PID is 825074 (from pkg/client/connector/service.go:573)
2021/06/15 19:19:07.7828 info    connector :  (from pkg/client/connector/service.go:574)
2021/06/15 19:19:08.0097 info    connector/background-init : Connecting to daemon... (from pkg/client/connector/service.go:407)
2021/06/15 19:19:08.0105 info    connector/background-init : Connecting to k8s cluster... (from pkg/client/connector/service.go:421)
2021/06/15 19:19:08.0259 info    connector/background-init : Server version v1.16.15+k3s1 (from pkg/client/connector/k8s_cluster.go:157)
2021/06/15 19:19:08.0259 info    connector/background-init : Context: k3d-single-zone-test (from pkg/client/connector/k8s_cluster.go:366)
2021/06/15 19:19:08.0259 info    connector/background-init : Server: https://0.0.0.0:33963 (from pkg/client/connector/k8s_cluster.go:367)
2021/06/15 19:19:08.0260 info    connector/background-init : Connected to context k3d-single-zone-test (https://0.0.0.0:33963) (from pkg/client/connector/service.go:440)
2021/06/15 19:19:08.0311 info    connector/background-init : [pid:825106] started command []string{"kubectl", "version", "--client", "-o", "json"}
2021/06/15 19:19:08.0314 info    connector/background-init : [pid:825106] stdin  < EOF
2021/06/15 19:19:08.1178 info    connector/background-init : [pid:825106] stdout > "{\n"
2021/06/15 19:19:08.1178 info    connector/background-init : [pid:825106] stdout > "  \"clientVersion\": {\n"
2021/06/15 19:19:08.1178 info    connector/background-init : [pid:825106] stdout > "    \"major\": \"1\",\n"
2021/06/15 19:19:08.1179 info    connector/background-init : [pid:825106] stdout > "    \"minor\": \"16\",\n"
2021/06/15 19:19:08.1179 info    connector/background-init : [pid:825106] stdout > "    \"gitVersion\": \"v1.16.15\",\n"
2021/06/15 19:19:08.1179 info    connector/background-init : [pid:825106] stdout > "    \"gitCommit\": \"2adc8d7091e89b6e3ca8d048140618ec89b39369\",\n"
2021/06/15 19:19:08.1179 info    connector/background-init : [pid:825106] stdout > "    \"gitTreeState\": \"clean\",\n"
2021/06/15 19:19:08.1179 info    connector/background-init : [pid:825106] stdout > "    \"buildDate\": \"2020-09-02T11:40:00Z\",\n"
2021/06/15 19:19:08.1179 info    connector/background-init : [pid:825106] stdout > "    \"goVersion\": \"go1.13.15\",\n"
2021/06/15 19:19:08.1179 info    connector/background-init : [pid:825106] stdout > "    \"compiler\": \"gc\",\n"
2021/06/15 19:19:08.1179 info    connector/background-init : [pid:825106] stdout > "    \"platform\": \"linux/amd64\"\n"
2021/06/15 19:19:08.1180 info    connector/background-init : [pid:825106] stdout > "  }\n"
2021/06/15 19:19:08.1180 info    connector/background-init : [pid:825106] stdout > "}\n"
2021/06/15 19:19:08.1192 info    connector/background-init : [pid:825106] finished successfully: exit status 0
2021/06/15 19:19:08.1193 info    connector/background-init : Connecting to traffic manager... (from pkg/client/connector/service.go:467)
2021/06/15 19:19:08.1193 info    connector/background-init : Waiting for TrafficManager to connect (from pkg/client/connector/service.go:482)
2021/06/15 19:19:08.1240 debug   connector/background-manager : Found Namespace ambassador. (from pkg/install/resource/instance.go:128)
2021/06/15 19:19:08.1257 debug   connector/background-manager : Found ServiceAccount traffic-manager.ambassador (from pkg/install/resource/instance.go:128)
2021/06/15 19:19:08.1375 debug   connector/background-manager : Found ClusterRole traffic-manager. (from pkg/install/resource/instance.go:128)
2021/06/15 19:19:08.1402 debug   connector/background-manager : Found ClusterRoleBinding traffic-manager. (from pkg/install/resource/instance.go:128)
2021/06/15 19:19:08.1419 debug   connector/background-manager : Found Secret mutator-webhook-tls.ambassador (from pkg/install/resource/instance.go:128)
2021/06/15 19:19:08.1448 debug   connector/background-manager : Found Deployment traffic-manager.ambassador (from pkg/install/resource/instance.go:128)
2021/06/15 19:19:08.1449 info    connector/background-manager : Deployment traffic-manager.ambassador is up-to-date. Image: docker.io/datawire/tel2:2.3.1 (from pkg/install/resource/tm_deployment.go:173)
2021/06/15 19:19:08.1474 debug   connector/background-manager : Found Service traffic-manager.ambassador (from pkg/install/resource/instance.go:128)
2021/06/15 19:19:08.1487 debug   connector/background-manager : Found Service agent-injector.ambassador (from pkg/install/resource/instance.go:128)
2021/06/15 19:19:08.2318 debug   connector/background-manager : Unable to find MutatingWebhookConfiguration agent-injector-webhook.ambassador (from pkg/install/resource/instance.go:125)
2021/06/15 19:19:08.2319 info    connector/background-manager : Creating MutatingWebhookConfiguration agent-injector-webhook.ambassador (from pkg/install/resource/instance.go:92)
2021/06/15 19:19:08.4308 error   connector/background-manager : goroutine "/connector/background-manager" exited with error: failed to create MutatingWebhookConfiguration agent-injector-webhook.ambassador: MutatingWebhookConfiguration.admissionregistration.k8s.io "agent-injector-webhook" is invalid: webhooks[0].admissionReviewVersions: Invalid value: []string{"v1"}: must include at least one of v1beta1
2021/06/15 19:19:08.4308 info    connector/server-grpc:shutdown_logger : shutting down (gracefully)...
2021/06/15 19:19:08.4309 info    connector/background-systema:shutdown_logger : shutting down (gracefully)...
2021/06/15 19:19:08.4309 error   connector/background-init : Failed to initialize session with traffic-manager: context canceled (from pkg/client/connector/service.go:486)
2021/06/15 19:19:08.4309 debug   connector/background-init : goroutine "/connector/background-init" exited without error
2021/06/15 19:19:08.4310 debug   connector/background-metriton : goroutine "/connector/background-metriton" exited without error
2021/06/15 19:19:08.4310 debug   connector/quit : goroutine "/connector/quit" exited without error
2021/06/15 19:19:08.4310 debug   connector/background-k8swatch : goroutine "/connector/background-k8swatch" exited without error
2021/06/15 19:19:08.4309 info    connector/server-grpc/GracefulStop : Shutting down (from pkg/client/connector/service.go:622)
2021/06/15 19:19:08.4309 debug   connector/background-systema/refresh : goroutine "/connector/background-systema/refresh" exited without error
2021/06/15 19:19:08.4309 info    connector:shutdown_logger : shutting down (gracefully)...
2021/06/15 19:19:08.4315 debug   connector/server-grpc/GracefulStop : goroutine "/connector/server-grpc/GracefulStop" exited without error
2021/06/15 19:19:08.4315 info    connector/server-grpc/Serve : Done serving (from pkg/client/connector/service.go:629)
2021/06/15 19:19:08.4316 debug   connector/server-grpc/Serve : goroutine "/connector/server-grpc/Serve" exited without error
2021/06/15 19:19:08.4316 debug   connector/server-grpc : Server ended (from pkg/client/connector/service.go:610)
2021/06/15 19:19:08.4316 debug   connector/server-grpc : goroutine "/connector/server-grpc" exited without error
2021/06/15 19:19:08.9310 debug   connector/background-systema/server-http : goroutine "/connector/background-systema/server-http" exited without error
2021/06/15 19:19:08.9311 debug   connector/background-systema : goroutine "/connector/background-systema" exited without error
2021/06/15 19:19:08.9312 info    connector:shutdown_status :   final goroutine statuses:
2021/06/15 19:19:08.9312 info    connector:shutdown_status :     /connector/background-init    : exited without error
2021/06/15 19:19:08.9312 info    connector:shutdown_status :     /connector/background-k8swatch: exited without error
2021/06/15 19:19:08.9313 info    connector:shutdown_status :     /connector/background-manager : exited with error
2021/06/15 19:19:08.9313 info    connector:shutdown_status :     /connector/background-metriton: exited without error
2021/06/15 19:19:08.9313 info    connector:shutdown_status :     /connector/background-systema : exited without error
2021/06/15 19:19:08.9313 info    connector:shutdown_status :     /connector/quit               : exited without error
2021/06/15 19:19:08.9313 info    connector:shutdown_status :     /connector/server-grpc        : exited without error
2021/06/15 19:19:08.9314 error   connector : failed to create MutatingWebhookConfiguration agent-injector-webhook.ambassador: MutatingWebhookConfiguration.admissionregistration.k8s.io "agent-injector-webhook" is invalid: webhooks[0].admissionReviewVersions: Invalid value: []string{"v1"}: must include at least one of v1beta1 (from pkg/client/connector/service.go:695)
telepresence: error: failed to create MutatingWebhookConfiguration agent-injector-webhook.ambassador: MutatingWebhookConfiguration.admissionregistration.k8s.io "agent-injector-webhook" is invalid: webhooks[0].admissionReviewVersions: Invalid value: []string{"v1"}: must include at least one of v1beta1

[darinpp]

Adding v1beta1 to AdmissionReviewVersions seems to be fixing the connect. Haven't done much testing

git diff
diff --git a/pkg/install/resource/injector_webhook.go b/pkg/install/resource/injector_webhook.go
index 0104172c..35b3dd61 100644
--- a/pkg/install/resource/injector_webhook.go
+++ b/pkg/install/resource/injector_webhook.go
@@ -60,7 +60,7 @@ func (ri injectorWebhook) Create(ctx context.Context) error {
                        FailurePolicy:           &failurePolicy,
                        SideEffects:             &sideEffects,
                        TimeoutSeconds:          &timeoutSecs,
-                       AdmissionReviewVersions: []string{"v1"},
+                       AdmissionReviewVersions: []string{"v1", "v1beta1"},
                },
        }
        return create(ctx, mwc)

and then I get successful connection

./telepresence  connect
Launching Telepresence Daemon v2.3.2-6-gb347edee-1623862682 (api v3)
Connecting to traffic manager...
Connected to context k3d-single-zone-test (https://0.0.0.0:33963)

[ManigandanMS]

@darinpp looks a good hack, thanks. I made similar change in my locate code base. How do I build it locally and get binary version of telepresence?

[thallgren]

@ManigandanMS @darinpp What type of cluster are you on? What version?

[ManigandanMS]

Server version: Client Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.3-cisco", GitCommit:"cd619a4c9d3aa3e2b673e0a1a2e36a42a721f20c", GitTreeState:"clean", BuildDate:"2019-09-06T05:30:14Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.3-cisco", GitCommit:"cd619a4c9d3aa3e2b673e0a1a2e36a42a721f20c", GitTreeState:"clean", BuildDate:"2019-09-06T05:30:14Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

Client Version: Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"darwin/amd64"}

[thallgren]

Ok, that explains it. The v1 version was introduced in a 1.16 version when v1beta1 was deprecated. The v1beta1 will be removed in the coming 1.22 version so I guess we'll need to add it conditionally.

[justinfiore]

I have the same issue. My k8s cluster version:

Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.12-eks-31566f", GitCommit:"31566f851673e809d4d667b7235ed87587d37722", GitTreeState:"clean", BuildDate:"2020-10-20T23:25:14Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

[ManigandanMS]

Ok, that explains it. The v1 version was introduced in a 1.16 version when v1beta1 was deprecated. The v1beta1 will be removed in the coming 1.22 version so I guess we'll need to add it conditionally.

I have checked out the project and have this change on local laptop. How do I build and generate the artifact?

[LukeShu]

@ManigandanMS wrote:

I have checked out the project and have this change on local laptop. How do I build and generate the artifact?

make build will generate the ./build-output/bin/telepresence binary. make push-image will build and push a ${TELEPRESENCE_REGISTRY}/tel2 Docker image. You should probably run them together as TELEPRESENCE_REGISTRY=… make build push-image so that the version numbers in the binary and the Docker image match.

There's more info in DEVELOPING.md.

[onmomo]

@LukeShu Any chance to include this fix in the pre-release?

[advance512]

We are limited to using microk8s based on v1.13 and will not be able to update for a few months. Because of this bug, we cannot use v2 of telepresence. Any chance this will be fixed soon? seems like a small patch.

[pencilkill]

+1

[lixajh]

same

[cindymullins-dw]

Kubernetes 1.17 is no longer officially supported. You can upgrade to the latest version of Telepresence here.