Failed VM customization on cse template install

[sachinphogat]

Describe the bug

While executing cse template install command, its failing with "Failed VM Customization error"

Required Python version: >= 3.7.3 Installed Python version: 3.7.11 (default, Aug 29 2022, 18:09:28) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] Validating config file 'config.yaml' InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. Connected to vCloud Director (100.121.43.146:443) Connected to vCenter Server 'PRODUCTION-VC' as '[email protected]' (chnvprodvc01.chnvsblab.com) Config file 'config.yaml' is valid Installing template 'ubuntu-20.04_k8-1.23_weave-2.8.1' at revision '*' on vCloud Director using config file 'config.yaml' Connected to vCD as system administrator: 100.121.43.146:443 Downloaded remote template cookbook from https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cust.sh' (file already exists) Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/init.sh' (file already exists) Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/nfsd.sh' (file already exists) Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/docker-upgrade.sh' (file already exists) Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/control-plane-cni-apply.sh' (file already exists) Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/control-plane-k8s-upgrade.sh' (file already exists) Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/worker-k8s-upgrade.sh' (file already exists) Found ova file 'ubuntu-20.04-server-cloudimg-amd64.ova' in catalog 'cse' Deleting temporary vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp' Deleted temporary vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp' Creating vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp' Found data file: /root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/init.sh Created vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp' Customizing vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp', vm 'ubuntu-2004-k8s1233-weave281-vm' Found data file: /root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cust.sh Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsNotRunning Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsNotRunning Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsNotRunning Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsRunning Failed VM customization. Check CSE install log Template Installation Error. Check CSE install logs Error while uploading file: 400

cse-install.log 22-08-30 15:30:30 | remote_template_manager:269 - get_unfiltered_remote_template_cookbook | DEBUG :: Downloaded remote template cookbook from https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml 22-08-30 15:30:30 | remote_template_manager:279 - get_unfiltered_remote_template_cookbook | DEBUG :: Template cookbook version: 2.0.0 22-08-30 15:30:30 | remote_template_manager:194 - _validate_remote_template_cookbook | DEBUG :: Template cookbook https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml is valid 22-08-30 15:30:31 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-20.04_k8-1.23_weave-2.8.1 revision 1 is supported by CSE 3.1.4 22-08-30 15:30:31 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.21_weave-2.8.1 revision 1 is supported by CSE 3.1.4 22-08-30 15:30:31 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.20_weave-2.6.5 revision 3 is supported by CSE 3.1.4 22-08-30 15:30:31 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.19_weave-2.6.5 revision 3 is supported by CSE 3.1.4 22-08-30 15:30:31 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.18_weave-2.6.5 revision 3 is supported by CSE 3.1.4 22-08-30 15:30:31 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template photon-v2_k8-1.14_weave-2.5.2 revision 5 is supported by CSE 3.1.4 22-08-30 15:30:31 | remote_template_manager:159 - _filter_unsupported_templates | DEBUG :: Successfully filtered unsupported templates. 22-08-30 15:30:32 | configure_cse:490 - install_template | INFO :: Installing template 'ubuntu-20.04_k8-1.23_weave-2.8.1' at revision '*' on vCloud Director using config file 'config.yaml' 22-08-30 15:30:32 | configure_cse:544 - install_template | INFO :: Connected to vCD as system administrator: 100.121.43.146:443 22-08-30 15:30:32 | remote_template_manager:269 - get_unfiltered_remote_template_cookbook | DEBUG :: Downloaded remote template cookbook from https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml 22-08-30 15:30:32 | remote_template_manager:279 - get_unfiltered_remote_template_cookbook | DEBUG :: Template cookbook version: 2.0.0 22-08-30 15:30:32 | remote_template_manager:194 - _validate_remote_template_cookbook | DEBUG :: Template cookbook https://raw.githubusercontent.com/vmware/container-service-extension-templates/master/template_v2.yaml is valid 22-08-30 15:30:32 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-20.04_k8-1.23_weave-2.8.1 revision 1 is supported by CSE 3.1.4 22-08-30 15:30:32 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.21_weave-2.8.1 revision 1 is supported by CSE 3.1.4 22-08-30 15:30:32 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.20_weave-2.6.5 revision 3 is supported by CSE 3.1.4 22-08-30 15:30:32 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.19_weave-2.6.5 revision 3 is supported by CSE 3.1.4 22-08-30 15:30:32 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template ubuntu-16.04_k8-1.18_weave-2.6.5 revision 3 is supported by CSE 3.1.4 22-08-30 15:30:32 | remote_template_manager:153 - _filter_unsupported_templates | DEBUG :: Template photon-v2_k8-1.14_weave-2.5.2 revision 5 is supported by CSE 3.1.4 22-08-30 15:30:32 | remote_template_manager:159 - _filter_unsupported_templates | DEBUG :: Successfully filtered unsupported templates. 22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cust.sh' (file already exists) 22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/init.sh' (file already exists) 22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/nfsd.sh' (file already exists) 22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/docker-upgrade.sh' (file already exists) 22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/control-plane-cni-apply.sh' (file already exists) 22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/control-plane-k8s-upgrade.sh' (file already exists) 22-08-30 15:30:32 | core_utils:286 - download_file | INFO :: Skipping download to '/root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cluster-upgrade/worker-k8s-upgrade.sh' (file already exists) 22-08-30 15:30:33 | template_builder:210 - _upload_source_ova | INFO :: Found ova file 'ubuntu-20.04-server-cloudimg-amd64.ova' in catalog 'cse' 22-08-30 15:30:33 | template_builder:188 - _delete_temp_vapp | INFO :: Deleting temporary vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp' 22-08-30 15:30:58 | template_builder:196 - _delete_temp_vapp | INFO :: Deleted temporary vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp' 22-08-30 15:30:58 | template_builder:258 - _create_temp_vapp | INFO :: Creating vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp' 22-08-30 15:30:58 | core_utils:331 - read_data_file | DEBUG :: Found data file: /root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/init.sh 22-08-30 15:31:40 | template_builder:286 - _create_temp_vapp | INFO :: Created vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp' 22-08-30 15:31:40 | template_builder:301 - _customize_vm | INFO :: Customizing vApp 'ubuntu-20.04_k8-1.23_weave-2.8.1_temp', vm 'ubuntu-2004-k8s1233-weave281-vm' 22-08-30 15:31:40 | core_utils:331 - read_data_file | DEBUG :: Found data file: /root/.cse_scripts/2.0.0/ubuntu-20.04_k8-1.23_weave-2.8.1_rev1/cust.sh 22-08-30 15:31:40 | vsphere_utils:74 - get_vsphere | DEBUG :: VM ID: urn:vcloud:vm:f1ba8786-a8f1-4667-8eb9-f5b4ce7270b4, Hostname: chnvprodvc01.chnvsblab.com 22-08-30 15:31:40 | vsphere_utils:100 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsNotRunning 22-08-30 15:31:46 | vsphere_utils:100 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsNotRunning 22-08-30 15:31:51 | vsphere_utils:100 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsNotRunning 22-08-30 15:31:56 | vsphere_utils:100 - callback | INFO :: Waiting for guest tools, status: "vm='vim.VirtualMachine:vm-9095', status=guestToolsRunning 22-08-30 15:33:11 | template_builder:351 - _customize_vm | ERROR :: Failed VM customization with error: Error while uploading file: 400 Traceback (most recent call last): File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/container_service_extension/installer/templates/template_builder.py", line 343, in _customize_vm msg_update_callback=self.msg_update_callback)) File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/vsphere_guest_run/vsphere.py", line 203, in execute_script_in_guest self.upload_file_to_guest(vm, user, password, content, target) File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/vsphere_guest_run/vsphere.py", line 147, in upload_file_to_guest 'Error while uploading file: %s' % resp.status_code) Exception: Error while uploading file: 400 22-08-30 15:33:11 | configure_cse:633 - install_template | ERROR :: Template Installation Error. Check CSE install logs Traceback (most recent call last): File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/container_service_extension/installer/configure_cse.py", line 599, in install_template msg_update_callback=msg_update_callback) File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/container_service_extension/installer/configure_cse.py", line 2201, in _install_single_template retain_temp_vapp=retain_temp_vapp File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/container_service_extension/installer/templates/template_builder.py", line 491, in build self._customize_vm(vapp, self.temp_vm_name) File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/container_service_extension/installer/templates/template_builder.py", line 343, in _customize_vm msg_update_callback=self.msg_update_callback)) File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/vsphere_guest_run/vsphere.py", line 203, in execute_script_in_guest self.upload_file_to_guest(vm, user, password, content, target) File "/var/sachin/Python-3.7.11/cse/lib/python3.7/site-packages/vsphere_guest_run/vsphere.py", line 147, in upload_file_to_guest 'Error while uploading file: %s' % resp.status_code) Exception: Error while uploading file: 400

Reproduction steps

Executing below command as part of cse installation guide(https://cloudsolutions.vmware.com/content/dam/digitalmarketing/microsites/en/images/cloud-solutions/pdfs/CSE_CDS_DeploymentGuide.pdf).
 cse template install ubuntu-20.04_k8-1.23_weave-2.8.1 -s

Expected behavior

.

Additional context

No response