Cluster creation fails : While initializing cluster

[Vijendrasi]

VCD version : 9.5 CSE, Container Service Extension for VMware vCloud Director, version 1.2.7 vcd-cli, VMware vCloud Director Command Line Interface, 21.1.0 Template : photon-custom-hw11-2.0-304b817-k8s

root@photon26156-001 [ /etc/systemd/system ]# vcd cse cluster create CSECLU01 --network ORG-NET-EXT01 --nodes 1 create_cluster: Creating cluster CSECLU01(0639dce8-c60b-4486-bbad-fa90dcaed74f) create_cluster: Creating cluster vApp CSECLU01(0639dce8-c60b-4486-bbad-fa90dcaed74f) create_cluster: Creating master node for CSECLU01(0639dce8-c60b-4486-bbad-fa90dcaed74f) create_cluster: Initializing cluster CSECLU01(0639dce8-c60b-4486-bbad-fa90dcaed74f), status: running -

CSE error logs : 19-05-23 03:46:04 | cluster:231 - wait_for_guest_execution_callback | DEBUG :: process [1, <Response [200]>, <Response [200]>] on vm 'vim.VirtualMachine:vm-208' finished, exit code: 1 19-05-23 03:46:04 | cluster:390 - execute_script_in_nodes | DEBUG :: 1 19-05-23 03:46:04 | cluster:391 - execute_script_in_nodes | DEBUG :: [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.06.0-ce. Max validated version: 17.03 [WARNING FileExisting-crictl]: crictl not found in system path Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl couldn't initialize a Kubernetes cluster

19-05-23 03:46:04 | cluster:392 - execute_script_in_nodes | DEBUG :: 19-05-23 03:46:04 | broker:526 - create_cluster_thread | ERROR :: Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/container_service_extension/broker.py", line 479, in create_cluster_thread init_cluster(server_config, vapp, template) File "/usr/lib/python3.7/site-packages/container_service_extension/cluster.py", line 285, in init_cluster f"Couldn't initialize cluster:\n{result[0][2].content.decode()}") container_service_extension.exceptions.ClusterInitializationError: Couldn't initialize cluster: [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.06.0-ce. Max validated version: 17.03 [WARNING FileExisting-crictl]: crictl not found in system path Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl couldn't initialize a Kubernetes cluster

[Vijendrasi]

root@photon26156-001 [ /etc/systemd/system ]# vcd cse template list catalog catalog_item description is_default name

---

cse photon-custom-hw11-2.0-304b817-k8s PhotonOS v2 True photon-v2 Docker 17.06.0-4 Kubernetes 1.9.1 weave 2.3.0 cse ubuntu-16.04-server-cloudimg-amd64-k8s Ubuntu 16.04 False ubuntu-16.04 Docker 18.03.0~ce Kubernetes 1.10.1 weave 2.3.0

[sahithi]

Hi Vijendrasi,

Are your templates updated after you have upgraded to CSE 1.2.7? I see a different version of docker on Release Notes for CSE 1.2.7. Is this issue happening only on Photon? Are you able to successfully create clusters with Ubuntu templates?

---

Next time when you attempt to deploy a cluster, run it with the option --disable-rollback. And try to run https://github.com/vmware/container-service-extension/blob/master/scripts/mstr-photon-v2.sh directly on the master node and paste the response here for each command in there.

[Vijendrasi]

I am using fresh deployment for CSE 1.2.7 I am facing another issue while Ubuntu template creation. Customization completes after that it fails.

---

upgrading the system customization completed

Customized vApp 'ubuntu' Creating template 'ubuntu-16.04-server-cloudimg-amd64-k8s' from vApp 'ubuntu' CSE Installation Error. Check CSE install logs Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/container_service_extension/configure_cse.py", line 695, in install_cse org=org) File "/usr/lib/python3.6/site-packages/container_service_extension/configure_cse.py", line 818, in c reate_template power_on=not template_config['cleanup']) File "/usr/lib/python3.6/site-packages/container_service_extension/configure_cse.py", line 1033, incapture_vapp_to_template catalog = org.get_catalog(catalog_name) File "/root/.local/lib/python3.6/site-packages/pyvcloud/vcd/org.py", line 177, in get_catalog return self.client.get_resource(href) File "/root/.local/lib/python3.6/site-packages/pyvcloud/vcd/client.py", line 1246, in get_resource 'GET', uri, objectify_results=objectify_results, params=params) File "/root/.local/lib/python3.6/site-packages/pyvcloud/vcd/client.py", line 982, in _do_request _objectify_response(response, objectify_results)) File "/root/.local/lib/python3.6/site-packages/pyvcloud/vcd/client.py", line 990, in response_code to_exception raise UnauthorizedException(sc, request_id, objectify_response) pyvcloud.vcd.exceptions.UnauthorizedException: Status code: 401, (request id: b0936ca8-da2b-4b01-b449-cb0a9d5e5e77)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/bin/cse", line 11, in sys.exit(cli()) File "/root/.local/lib/python3.6/site-packages/click/core.py", line 764, in call return self.main(*args, **kwargs) File "/root/.local/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/root/.local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/root/.local/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/root/.local/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/root/.local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/lib/python3.6/site-packages/container_service_extension/cse.py", line 239, in install ext_install=ext_install) File "/usr/lib/python3.6/site-packages/container_service_extension/configure_cse.py", line 702, in install_cse client.logout() File "/root/.local/lib/python3.6/site-packages/pyvcloud/vcd/client.py", line 940, in logout result = self._do_request('DELETE', uri) File "/root/.local/lib/python3.6/site-packages/pyvcloud/vcd/client.py", line 982, in _do_request _objectify_response(response, objectify_results)) File "/root/.local/lib/python3.6/site-packages/pyvcloud/vcd/client.py", line 990, in _response_code_to_exception raise UnauthorizedException(sc, request_id, objectify_response) pyvcloud.vcd.exceptions.UnauthorizedException: Status code: 401, (request id: 2b08ec3e-bdd6-4c13-88ee-405defea96e6)

[sahithi]

Hi,

Whom are you logged in as? Does user has enough permissions? It is clearly an "Unauthorized Exception".

For future reference, please also refer to this Known Issue - "Sharing CSE catalog with non organization administrator users" section in https://vmware.github.io/container-service-extension/CSE_ADMIN.html

[Vijendrasi]

Hi Sahithi, While running same command Photon k8s template successfully created and Ubuntu k8s template fails with this error. I tried this command with Org Admin and System admin as well.

With Photon k8s template I am unable to create cluster (error described above) so I am stuck photon and ubuntu templates have error in different stages.