Error on upgrading existing cse cluster in CSE 2.6.1

[sanjeevgorai]

Hello All, We have upgraded CSE 2.5.1 to CSE 2.6.1 in our environment and now trying to upgrade one of the existing cse cluster created from template photon-v2_k8-1.14_weave-2.5.2 rev 1 to photon-v2_k8-1.14_weave-2.5.2 rev 3 but getting below error messages.

please suggest.

[ /home/vmware ]# vcd cse cluster upgrade CLUSUPG01 photon-v2_k8-1.14_weave-2.5.2 3 cluster operation: Upgrading cluster 'CLUSUPG01' software to match template photon-v2_k8-1.14_weave-2.5.2 (revision 3): Kubernetes: 1.14.6 -> 1.14.10, Docker-CE: 18.06.cluster operation: Upgrading cluster 'CLUSUPG01' software to match template photon-v2_k8-1.14_weave-2.5.2 (revision 3): Kubernetes: 1.14.6 -> 1.14.10, Docker-CE: 18.06.2 -> 18.06.2-6, CNI: weave 2.5.2 -> 2.5.2 cluster operation: Draining master node ['mstr-0x7s'] cluster operation: Upgrading Kubernetes (1.14.6 -> 1.14.10) in master node ['mstr-0x7s'] task: 5dc1b207-d26b-43a5-84e1-9730b7bf7ce3, result: error, message: Unexpected error while upgrading cluster 'CLUSUPG01': Script execution failed on node ['mstr-0x7s'] Errors: ["curl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\nError: Failed to synchronize cache for repo 'VMware Photon Linux 2.0(x86_64) Updates' from 'https://dl.bintray.com/vmware/photon_updates_2.0_x86_64'\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\nError: Failed to synchronize cache for repo 'VMware Photon Linux 2.0(x86_64)' from 'https://dl.bintray.com/vmware/photon_release_2.0_x86_64'\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\nError: Failed to synchronize cache for repo 'VMware Photon Extras 2.0(x86_64)' from 'https://dl.bintray.com/vmware/photon_extras_2.0_x86_64'\nError: Failed to synchronize cache for repo 'VMware Photon Linux 2.0(x86_64) Updates' from 'https://dl.bintray.com/vmware/photon_updates_2.0_x86_64'\ncurl#6: Couldn't resolve host name\nError: Failed to synchronize cache for repo 'VMware Photon Linux 2.0(x86_64)' from 'https://dl.bintray.com/vmware/photon_release_2.0_x86_64'\nError: Failed to synchronize cache for repo 'VMware Photon Extras 2.0(x86_64)' from 'https://dl.bintray.com/vmware/photon_extras_2.0_x86_64'\nError(1011) : No matching packages\n"]

[Anirudh9794]

Hello, Photon templates stopped using bintray and moved to packages.vmware.com. This has now been fixed in the default template repository https://github.com/vmware/container-service-extension-templates/pull/19. Please delete ~/.cse-scripts folder and retry upgrade.

[sanjeevgorai]

Hello Anirudh

Thanks for your comment but still facing same issue after downloading new scripts in ~/.cse-scripts for new templates photon-v2_k8-1.14_weave-2.5.2 (revision 3)

vcd cse cluster upgrade ESA-CLS01 photon-v2_k8-1.14_weave-2.5.2 3 cluster operation: Upgrading cluster 'ESA-CLS01' software to match template photon-v2_k8-1.14_weave-2.5.2 (revision 3): Kubernetes: 1.14.6 -> 1.14.10, Docker-CE: 18.06.cluster operation: Upgrading cluster 'ESA-CLS01' software to match template photon-v2_k8-1.14_weave-2.5.2 (revision 3): Kubernetes: 1.14.6 -> 1.14.10, Docker-CE: 18.06.2 -> 18.06.2-6, CNI: weave 2.5.2 -> 2.5.2 cluster operation: Draining master node ['mstr-3y3r'] cluster operation: Upgrading Kubernetes (1.14.6 -> 1.14.10) in master node ['mstr-3y3r'] task: 6a86bdb7-d7f6-4664-911e-917910df0260, result: error, message: Unexpected error while upgrading cluster 'ESA-CLS01': Script execution failed on node ['mstr-3y3r'] Errors: ["curl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\ncurl#6: Couldn't resolve host name\nError(1207) : Unknown error\n"]

[rocknes]

Hi Sanjeev,

Can you please log into the vm and make sure that default dns has been configured on the machine, and the machine has internet connectivity? The error message seems to indicate that the vm either doesn't have internet access (hence unable to reach out to 8.8.8.8) or is being firewalled out.

Generally this class of errors stem from some form of misconfiguration of the org vdc network or restrictive nature of the network.

Regards Aritra Sen

[sanjeevgorai]

Hello Aritra

Thanks for your comment. We don't have direct connection in production environment but internet access is allowed using proxy server and internet connection is working fine as we are able to download templates and script from vmware github page. But facing this issue while upgrading any existing cluster from one rev to another. Might be some urls are not allowed in proxy setting.

Is there any list of urls that is being used by CSE so that I can request to open these urls in proxy settings.

[rocknes]

Unfortunately we don't have a handy list of URLs that the scripts reach out to. This is so because, CSE just calls kubeadm and has no clue regarding the urls the command would reach out to. CSE by itself just reaches out to the github template repo, which I guess is accessible in this case.

You can try to parse the cse server debug logs to get a sense of the blocked urls and add them to the whitelist one at a time. I agree It is a cumbersome process, but it's the best that can be done at this point.

Regards Aritra Sen