eks-anywhere icon indicating copy to clipboard operation
eks-anywhere copied to clipboard

Published 20 hours ago • verified publisher icon aws

Proxy Config not loading for Workload Cluster from Bootstrap Cluster

Open vipulsha22 opened this issue 2 years ago • 6 comments

What happened: Cluster creation failed . Proxy config not updated in workload cluster.

What you expected to happen: Successful creation of cluster

How to reproduce it (as minimally and precisely as possible): Create a cluster using proxy

Anything else we need to know?: Updated @ https://github.com/aws/eks-anywhere/issues/2240#issuecomment-1157401678

Environment: Local Cluster

  • EKS Anywhere Release: v0.9.1

vipulsha22 avatar May 24 '22 15:05 vipulsha22

Any chance when running the command directly with the insecure flag there is more output as to the issue?

jaxesn avatar May 26 '22 03:05 jaxesn

It's giving i/o timeout.

[xxx@XXXX ~]# docker exec -i -e HELM_EXPERIMENTAL_OCI=1 -e HTTP_PROXY=10.90.xxx.xxx:xxxx -e HTTPS_PROXY=10.90.xxx.xxx:xxxx eksa_1653462201850010822 helm template oci://public.ecr.aws/isovalent/cilium --version 1.9.13-eksa.2 --insecure-skip-tls-verify --namespace kube-system --debug
install.go:178: [debug] Original chart version: "1.9.13-eksa.2"
Error: failed to do request: Head "https://public.ecr.aws/v2/isovalent/cilium/manifests/1.9.13-eksa.2": dial tcp: lookup public.ecr.aws on 10.90.xxx.xxx:xxxx: read udp 10.90.xxx.xxx:xxxx->10.90.xxx.xxx:xxxx: i/o timeout
helm.go:84: [debug] Head "https://public.ecr.aws/v2/isovalent/cilium/manifests/1.9.13-eksa.2": dial tcp: lookup public.ecr.aws on 10.90.xxx.xxx:xxxx: read udp 10.90.xxx.xxx:xxxx->10.90.xxx.xxx:xxxx: i/o timeout
failed to do request
github.com/containerd/containerd/remotes/docker.(*request).do
        github.com/containerd/[email protected]/remotes/docker/resolver.go:569
github.com/containerd/containerd/remotes/docker.(*request).doWithRetries
        github.com/containerd/[email protected]/remotes/docker/resolver.go:576
github.com/containerd/containerd/remotes/docker.(*dockerResolver).Resolve
        github.com/containerd/[email protected]/remotes/docker/resolver.go:280
oras.land/oras-go/pkg/oras.Copy
        oras.land/[email protected]/pkg/oras/copy.go:63
helm.sh/helm/v3/pkg/registry.(*Client).Pull
        helm.sh/helm/v3/pkg/registry/client.go:322
helm.sh/helm/v3/pkg/getter.(*OCIGetter).get
        helm.sh/helm/v3/pkg/getter/ocigetter.go:53
helm.sh/helm/v3/pkg/getter.(*OCIGetter).Get
        helm.sh/helm/v3/pkg/getter/ocigetter.go:36
helm.sh/helm/v3/pkg/downloader.(*ChartDownloader).DownloadTo
        helm.sh/helm/v3/pkg/downloader/chart_downloader.go:100
helm.sh/helm/v3/pkg/action.(*ChartPathOptions).LocateChart
        helm.sh/helm/v3/pkg/action/install.go:760
main.runInstall
        helm.sh/helm/v3/cmd/helm/install.go:190
main.newTemplateCmd.func2
        helm.sh/helm/v3/cmd/helm/template.go:82
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:974
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:902
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_amd64.s:1581

vipulsha22 avatar May 26 '22 04:05 vipulsha22

That looks like a DNS not resolving issue

rimaulana avatar Jun 15 '22 21:06 rimaulana

Somehow my proxy config is getting updated in 'dev-cluster-eks-a-cluster-control-plane' container only.

[xx@xx xx]# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5e4bafcd71c3 public.ecr.aws/eks-anywhere/kubernetes-sigs/kind/node:v1.22.6-eks-d-1-22-6-eks-a-10 "/usr/local/bin/entr…" 12 minutes ago Up 12 minutes dev-cluster-md-0-7858f8695f-dwxnc aa6409a73e5b public.ecr.aws/eks-anywhere/kubernetes-sigs/kind/node:v1.22.6-eks-d-1-22-6-eks-a-10 "/usr/local/bin/entr…" 14 minutes ago Up 14 minutes 40645/tcp, 127.0.0.1:40645->6443/tcp dev-cluster-mpxzx e38931121cd9 public.ecr.aws/eks-anywhere/kubernetes-sigs/kind/node:v1.22.6-eks-d-1-22-6-eks-a-10 "/usr/local/bin/entr…" 14 minutes ago Up 14 minutes dev-cluster-etcd-8rfkf 1a6b2d38fc99 public.ecr.aws/eks-anywhere/kubernetes-sigs/kind/haproxy:v0.12.0-eks-a-10 "haproxy -sf 7 -W -d…" 14 minutes ago Up 14 minutes 40423/tcp, 0.0.0.0:40423->6443/tcp dev-cluster-lb b5911606e56a public.ecr.aws/eks-anywhere/kubernetes-sigs/kind/node:v1.22.6-eks-d-1-22-6-eks-a-10 "/usr/local/bin/entr…" 17 minutes ago Up 17 minutes 127.0.0.1:46271->6443/tcp dev-cluster-eks-a-cluster-control-plane

Installing cilium using helm install -f helmtest.yaml cilium oci://public.ecr.aws/isovalent/cilium --version "1.9.13-eksa.2" -n kube-system (extracted helm template manually in helmtest.yaml file)

I am manually updating proxy config in dev-cluster-mpxzx and dev-cluster-md-0-7858f8695f-dwxnc (only then coredns and cilium pods are coming in running status) using https://stackoverflow.com/questions/72004749/eks-anywhere-cluster-cert-manager-io-timeout

[xx@xxxx]# helm install -f helmtest.yaml cilium oci://public.ecr.aws/isovalent/cilium --version "1.9.13-eksa.2" -n kube-system W0616 13:47:03.622379 12742 warnings.go:70] spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead NAME: cilium LAST DEPLOYED: Thu Jun 16 13:47:02 2022 NAMESPACE: kube-system STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: You have successfully installed Cilium with Hubble. Your release version is 1.9.13-eksa.2. For any further help, visit https://docs.cilium.io/en/v1.9/gettinghelp

NAMESPACE NAME READY STATUS RESTARTS AGE kube-system cilium-2k6z7 1/1 Running 0 3m22s kube-system cilium-gl7qx 1/1 Running 0 3m13s kube-system cilium-operator-5898956498-4dfw5 1/1 Running 0 3m22s kube-system cilium-operator-5898956498-zkcdq 1/1 Running 0 3m22s kube-system coredns-55467bc785-54qsm 1/1 Running 0 4m4s kube-system coredns-55467bc785-b7mxs 1/1 Running 0 4m4s kube-system kube-apiserver-dev-cluster-mpxzx 1/1 Running 0 3m58s kube-system kube-controller-manager-dev-cluster-mpxzx 1/1 Running 0 3m58s kube-system kube-proxy-fzg7z 1/1 Running 0 4m4s kube-system kube-proxy-qbt44 1/1 Running 0 3m13s kube-system kube-scheduler-dev-cluster-mpxzx 1/1 Running 0 3m58s

Cluster create is still getting same error 2022-06-16T13:47:13.132+0530 V6 Executing command {"cmd": "/usr/bin/docker exec -i -e NO_PROXY=192.168.0.0/16,10.96.0.0/12 -e HELM_EXPERIMENTAL_OCI=1 -e HTTP_PROXY=10.90.xxx.xxx:xxxx -e HTTPS_PROXY=10.90.xxx.xxx:xxxx eksa_1655367048729034146 helm template oci://public.ecr.aws/isovalent/cilium --version 1.9.13-eksa.2 --insecure-skip-tls-verify --namespace kube-system -f -"} 2022-06-16T13:47:23.241+0530 V9 docker {"stderr": "Error: failed to download \"oci://public.ecr.aws/isovalent/cilium\" at version \"1.9.13-eksa.2\"\n"}

And finally failing Error: generating networking manifest: failed generating cilium manifest: Error: failed to download "oci://public.ecr.aws/isovalent/cilium" at version "1.9.13-eksa.2"

Now ,no idea how to proceed further. I am trying EKS Anywhere (on anything related to cloud) for first time. :) Any help is highly appreciated.

vipulsha22 avatar Jun 16 '22 08:06 vipulsha22

Same for me.

docker run -d --name eksa_1655969613989358333 --network host --entrypoint sleep public.ecr.aws/eks-anywhere/cli-tools:v0.7.2-eks-a-10 infinity

docker exec -i -e HELM_EXPERIMENTAL_OCI=1 -e HTTP_PROXY=10.192.xxx.xxx:3128 -e HTTPS_PROXY=10.192.xxx.xxx:3128 eksa_1655969613989358333 helm template oci://public.ecr.aws/isovalent/cilium --version 1.9.13-eksa.2 --insecure-skip-tls-verify --namespace kube-system --debug

install.go:178: [debug] Original chart version: "1.9.13-eksa.2"
Error: failed to do request: Head "https://public.ecr.aws/v2/isovalent/cilium/manifests/1.9.13-eksa.2": dial tcp 99.83.145.10:443: connect: connection timed out
helm.go:84: [debug] Head "https://public.ecr.aws/v2/isovalent/cilium/manifests/1.9.13-eksa.2": dial tcp 99.83.145.10:443: connect: connection timed out
failed to do request
github.com/containerd/containerd/remotes/docker.(*request).do
	github.com/containerd/[email protected]/remotes/docker/resolver.go:569
github.com/containerd/containerd/remotes/docker.(*request).doWithRetries
	github.com/containerd/[email protected]/remotes/docker/resolver.go:576
github.com/containerd/containerd/remotes/docker.(*dockerResolver).Resolve
	github.com/containerd/[email protected]/remotes/docker/resolver.go:280
oras.land/oras-go/pkg/oras.Copy
	oras.land/[email protected]/pkg/oras/copy.go:63
helm.sh/helm/v3/pkg/registry.(*Client).Pull
	helm.sh/helm/v3/pkg/registry/client.go:322
helm.sh/helm/v3/pkg/getter.(*OCIGetter).get
	helm.sh/helm/v3/pkg/getter/ocigetter.go:53
helm.sh/helm/v3/pkg/getter.(*OCIGetter).Get
	helm.sh/helm/v3/pkg/getter/ocigetter.go:36
helm.sh/helm/v3/pkg/downloader.(*ChartDownloader).DownloadTo
	helm.sh/helm/v3/pkg/downloader/chart_downloader.go:100
helm.sh/helm/v3/pkg/action.(*ChartPathOptions).LocateChart
	helm.sh/helm/v3/pkg/action/install.go:760
main.runInstall
	helm.sh/helm/v3/cmd/helm/install.go:190
main.newTemplateCmd.func2
	helm.sh/helm/v3/cmd/helm/template.go:82
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/[email protected]/command.go:974
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:902
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:255
runtime.goexit
	runtime/asm_amd64.s:1581

But when I manually use the cli-tools:v0.7.2-eks-a-8 container, I get the helm chart. With the cli-tools:v0.7.2-eks-a-9 container it does not work for me either.

docker run -d --name eksa_1655969613989358333 --network host --entrypoint sleep public.ecr.aws/eks-anywhere/cli-tools:v0.7.2-eks-a-8 infinity

docker exec -i -e HELM_EXPERIMENTAL_OCI=1 -e HTTP_PROXY=10.192.xxx.xxx:3128 -e HTTPS_PROXY=10.192.xxx.xxx:3128 eksa_1655969613989358333 helm template oci://public.ecr.aws/isovalent/cilium --version 1.9.13-eksa.2 --insecure-skip-tls-verify --namespace kube-system --debug

install.go:178: [debug] Original chart version: "1.9.13-eksa.2"
install.go:199: [debug] CHART PATH: /root/.cache/helm/repository/cilium-1.9.13-eksa.2.tgz

---
# Source: cilium/templates/cilium-agent-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cilium
  namespace: kube-system
---
# Source: cilium/templates/cilium-operator-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cilium-operator
  namespace: kube-system
---
# Source: cilium/templates/cilium-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-config
  namespace: kube-system
data:
...

janre avatar Jun 23 '22 14:06 janre

I can also confirm using same config/proxy , cluster is getting created successfully with v0.8.2 version.

2022-06-24T12:38:57.224+0530    V0      🎉 Cluster created!
2022-06-24T12:38:57.224+0530    V4      Task finished   {"task_name": "delete-kind-cluster", "duration": "4.179741254s"}
2022-06-24T12:38:57.224+0530    V4      ----------------------------------
2022-06-24T12:38:57.224+0530    V4      Tasks completed {"duration": "10m5.274211317s"}
2022-06-24T12:38:57.224+0530    V3      Cleaning up long running container      {"name": "eksa_1656053888653166077"}
2022-06-24T12:38:57.225+0530    V6      Executing command       {"cmd": "/usr/bin/docker rm -f -v eksa_1656053888653166077"}
[xxxx@xxxx]# eksctl anywhere version
v0.8.2
[xxxx@xxxx]#

vipulsha22 avatar Jun 24 '22 07:06 vipulsha22

Closing due to inactivity for over 60 days. @vipulsha22 have you had a chance to try a newer version? If this is still an issue for you we can reopen this issue.

drewvanstone avatar Nov 17 '22 22:11 drewvanstone