Error deploying KubeCluster as Tenant Administrator

[CrazyVolnay]

Hi,

I'm new and still learning to use CSE :) Everything is up to date and fresh install : VSphere 7.0.2 VCD 10.3 CSE, Container Service Extension for VMware vCloud Director, version 3.1.0

I've been able to setup and deploy Kube Clusters within the Organization / VDC definied in the config file loggued as Tnant Organization Admin. I can also deploy Kube Cluster in any Organization / VDC loggued as system admin. But I cannot deploy Kube Cluster loggued as a tenant organization admin. I've note the user had to have the right 'Catalog: View Published Catalogs', which is not present in the Organization Administrator. Instead I have 'View Private and Shared Catalogs within Current Organization' and 'View Shared Catalogs from Other Organizations' :

When I reach Kubernetes Container Clusters loggued as admin tenant, I first receive an uncommon error error :

And when I end the wizzard I receive the following error :

In server debug log, here the event thrown when creating the cluster :

21-10-08 14:04:56 | request_dispatcher:846 - process_request | Request Id: 2e196f03-beec-4be4-ba9b-3a0c93ff5d10 | DEBUG :: Incoming request message: {"id": "75e69641-dcc1-4195-a557-4c43a932f7bf", "method": "POST", "requestUri": "/api/cse/3.0/clusters", "queryString": null, "protocol": "HTTP/1.1", "scheme": "https", "remoteAddr": "<CLIENT_IP>", "remotePort": 15019, "localAddr": "<VCLOUD_IP>", "localPort": 443, "headers": {"Origin": "https://<VCLOUD_URL>", "Cookie": "_pk_ses.1.4192=0; vcloud_jwt=eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbmRlbW9AbmV0aXdhbi5mciIsImlzcyI6ImMzMDgxZmY2LTQ2MWMtNDU0MS1hMWQ5LTQwMmUxMzM5N2UyY0BkMDZjYjMzOS0zNGQ2LTQ5OWEtOWY5NS02ZTA2MDI1M2E0NjQiLCJleHAiOjE2MzM3ODgxODUsInZlcnNpb24iOiJ2Y2xvdWRfMS4wIiwianRpIjoiOGFjMTJlZWM3ODBhNDFkYzgwMjMwMTk2Mzg2NDNlZmUifQ.OYu9rp6szwv4Kjw6flkvpH4Wi2zIQGMpycFnr7g_Tl_rUswVjW6Cuyxs0fmLgbYKyfLd1pmkJO-3nSUGwgCD60EsvB3tIhGxeXFunx-SpsX3bvp-XmM6YuiYQbOnF6ZSO4souo1EpID_63hVx5fH2-xLFaka65_q_FMxfY_MGdwc7Ex8Em5Cw1BuDeWBSw41_kO8kXg5ZKyzMmpKa4okcsJStOnrCWdg-YK6iRTq4o4Zxori69h4u_DiQys8fxzSEmOPVmWlAiYUXq7Z76LtjdaLGdTvAAkQ55Z0qatz26hqaXeeLfENP1h7CKroYZE0Jp64gG0cVMiqbOL6Ck-o2g; vcloud_session_id=8ac12eec780a41dc8023019638643efe; _pk_id.1.4192=5fd32b7fe35e233a.1600335786.40.1633701818.1633697885..d66c0e742a78a02c4e2dd63b6bab52e45692779d79c2d5e812e295eb7eee3cbe", "Accept": "application/+json;version=36.0, application/json;version=36.0", "Connection": "keep-alive", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36", "Referer": "https://<VCLOUD_URL>/tenant/demo_org/plugins/Vk13YXJl/cse", "Sec-Fetch-Site": "same-origin", "Sec-Fetch-Dest": "empty", "Host": "<VCLOUD_URL>", "Accept-Encoding": "gzip, deflate, br", "Sec-Fetch-Mode": "cors", "Authorization": [[REDACTED], "sec-ch-ua": ""Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"", "sec-ch-ua-mobile": "?0", "sec-ch-ua-platform": ""Windows"", "Accept-Language": "en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7", "Content-Length": "580", "Content-Type": "application/+json"}, "body": "eyJhcGlWZXJzaW9uIjoiY3NlLnZtd2FyZS5jb20vdjIuMCIsImtpbmQiOiJuYXRpdmUiLCJtZXRhZGF0YSI6eyJhZGRpdGlvbmFsUHJvcGVydGllcyI6dHJ1ZSwib3JnTmFtZSI6IkRFTU9fT1JHIiwidmlydHVhbERhdGFDZW50ZXJOYW1lIjoiREVNT19WREMiLCJuYW1lIjoiemRhZHphemRhemRhZHphIiwic2l0ZSI6IiJ9LCJzcGVjIjp7ImFkZGl0aW9uYWxQcm9wZXJ0aWVzIjp0cnVlLCJ0b3BvbG9neSI6eyJjb250cm9sUGxhbmUiOnsiY291bnQiOjEsInNpemluZ0NsYXNzIjoiTCIsInN0b3JhZ2VQcm9maWxlIjoiUkFJRDUifSwid29ya2VycyI6eyJjb3VudCI6Miwic3RvcmFnZVByb2ZpbGUiOiJSQUlENSJ9LCJuZnMiOnsiY291bnQiOjAsInNpemluZ0NsYXNzIjpudWxsLCJzdG9yYWdlUHJvZmlsZSI6bnVsbH19LCJzZXR0aW5ncyI6eyJvdmRjTmV0d29yayI6IkRFTU8tSUFBUy1MQU4iLCJzc2hLZXkiOm51bGwsInJvbGxiYWNrT25GYWlsdXJlIjp0cnVlfSwiZGlzdHJpYnV0aW9uIjp7InRlbXBsYXRlTmFtZSI6InVidW50dS0xNi4wNF9rOC0xLjE4X3dlYXZlLTIuNi41IiwidGVtcGxhdGVSZXZpc2lvbiI6Mn19fQ==", "statusCode": 0, "request": true} 21-10-08 14:04:56 | request_dispatcher:972 - process_request | Request Id: 2e196f03-beec-4be4-ba9b-3a0c93ff5d10 | DEBUG :: request body: {'apiVersion': 'cse.vmware.com/v2.0', 'kind': 'native', 'metadata': {'additionalProperties': True, 'orgName': 'DEMO_ORG', 'virtualDataCenterName': 'DEMO_VDC', 'name': 'zdadzazdazdadza', 'site': ''}, 'spec': {'additionalProperties': True, 'topology': {'controlPlane': {'count': 1, 'sizingClass': 'L', 'storageProfile': 'RAID5'}, 'workers': {'count': 2, 'storageProfile': 'RAID5'}, 'nfs': {'count': 0, 'sizingClass': None, 'storageProfile': None}}, 'settings': {'ovdcNetwork': 'DEMO-IAAS-LAN', 'sshKey': None, 'rollbackOnFailure': True}, 'distribution': {'templateName': 'ubuntu-16.04_k8-1.18_weave-2.6.5', 'templateRevision': 2}}} 21-10-08 14:04:56 | entity_service:53 - exception_handler_wrapper | Request Id: 2e196f03-beec-4be4-ba9b-3a0c93ff5d10 | ERROR :: [ c45551bb-5dd2-496b-a205-4ea6d0b1f9a7 ] This operation is denied. 21-10-08 14:04:56 | request_utils:166 - exception_handler_wrapper | Request Id: 2e196f03-beec-4be4-ba9b-3a0c93ff5d10 | ERROR :: [ c45551bb-5dd2-496b-a205-4ea6d0b1f9a7 ] This operation is denied. 21-10-08 14:04:56 | exception_handler:53 - exception_handler_wrapper | Request Id: 2e196f03-beec-4be4-ba9b-3a0c93ff5d10 | ERROR :: Traceback (most recent call last): File "/opt/vmware/cse/python/lib/python3.7/site-packages/container_service_extension/rde/common/entity_service.py", line 49, in exception_handler_wrapper result = func(*args, **kwargs) File "/opt/vmware/cse/python/lib/python3.7/site-packages/container_service_extension/rde/common/entity_service.py", line 110, in create_entity return_response_headers=is_request_async) File "/opt/vmware/cse/python/lib/python3.7/site-packages/container_service_extension/lib/cloudapi/cloudapi_client.py", line 134, in do_request response.raise_for_status() File "/opt/vmware/cse/python/lib/python3.7/site-packages/requests/models.py", line 953, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://<VCLOUD_URL>/cloudapi/1.0.0/entityTypes/urn:vcloud:type:cse:nativeCluster:2.0.0

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/vmware/cse/python/lib/python3.7/site-packages/container_service_extension/exception/exception_handler.py", line 37, in exception_handler_wrapper result = func(*args, **kwargs) File "/opt/vmware/cse/python/lib/python3.7/site-packages/container_service_extension/server/request_dispatcher.py", line 993, in process_request body_content = handler_method(request_data, operation_ctx) File "/opt/vmware/cse/python/lib/python3.7/site-packages/container_service_extension/lib/telemetry/telemetry_handler.py", line 112, in wrapper raise err File "/opt/vmware/cse/python/lib/python3.7/site-packages/container_service_extension/lib/telemetry/telemetry_handler.py", line 106, in wrapper ret_value = func(*args, **kwargs) File "/opt/vmware/cse/python/lib/python3.7/site-packages/container_service_extension/server/request_handlers/request_utils.py", line 167, in exception_handler_wrapper raise error File "/opt/vmware/cse/python/lib/python3.7/site-packages/container_service_extension/server/request_handlers/request_utils.py", line 161, in exception_handler_wrapper result = func(*args, **kwargs) File "/opt/vmware/cse/python/lib/python3.7/site-packages/container_service_extension/server/request_handlers/cluster_handler.py", line 85, in cluster_create is_request_async=True) File "/opt/vmware/cse/python/lib/python3.7/site-packages/container_service_extension/rde/common/entity_service.py", line 56, in exception_handler_wrapper minor_error_code=MinorErrorCode.DEFAULT_ERROR_CODE) container_service_extension.exception.exceptions.DefEntityServiceError: [ c45551bb-5dd2-496b-a205-4ea6d0b1f9a7 ] This operation is denied.

21-10-08 14:04:56 | mqtt_consumer:73 - process_mqtt_message | Request Id: 2e196f03-beec-4be4-ba9b-3a0c93ff5d10 | DEBUG :: Received message with request_id: 2e196f03-beec-4be4-ba9b-3a0c93ff5d10, mid: 14, and msg json: {'id': '75e69641-dcc1-4195-a557-4c43a932f7bf', 'method': 'POST', 'requestUri': '/api/cse/3.0/clusters', 'queryString': None, 'protocol': 'HTTP/1.1', 'scheme': 'https', 'remoteAddr': '<CLIENT_IP>', 'remotePort': 15019, 'localAddr': '<VCLOUD_IP>', 'localPort': 443, 'headers': {'Origin': 'https://<VCLOUD_URL>', 'Cookie': '_pk_ses.1.4192=0; vcloud_jwt=eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbmRlbW9AbmV0aXdhbi5mciIsImlzcyI6ImMzMDgxZmY2LTQ2MWMtNDU0MS1hMWQ5LTQwMmUxMzM5N2UyY0BkMDZjYjMzOS0zNGQ2LTQ5OWEtOWY5NS02ZTA2MDI1M2E0NjQiLCJleHAiOjE2MzM3ODgxODUsInZlcnNpb24iOiJ2Y2xvdWRfMS4wIiwianRpIjoiOGFjMTJlZWM3ODBhNDFkYzgwMjMwMTk2Mzg2NDNlZmUifQ.OYu9rp6szwv4Kjw6flkvpH4Wi2zIQGMpycFnr7g_Tl_rUswVjW6Cuyxs0fmLgbYKyfLd1pmkJO-3nSUGwgCD60EsvB3tIhGxeXFunx-SpsX3bvp-XmM6YuiYQbOnF6ZSO4souo1EpID_63hVx5fH2-xLFaka65_q_FMxfY_MGdwc7Ex8Em5Cw1BuDeWBSw41_kO8kXg5ZKyzMmpKa4okcsJStOnrCWdg-YK6iRTq4o4Zxori69h4u_DiQys8fxzSEmOPVmWlAiYUXq7Z76LtjdaLGdTvAAkQ55Z0qatz26hqaXeeLfENP1h7CKroYZE0Jp64gG0cVMiqbOL6Ck-o2g; vcloud_session_id=8ac12eec780a41dc8023019638643efe; _pk_id.1.4192=5fd32b7fe35e233a.1600335786.40.1633701818.1633697885..d66c0e742a78a02c4e2dd63b6bab52e45692779d79c2d5e812e295eb7eee3cbe', 'Accept': 'application/+json;version=36.0, application/json;version=36.0', 'Connection': 'keep-alive', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36', 'Referer': 'https://<VCLOUD_URL>/tenant/demo_org/plugins/Vk13YXJl/cse', 'Sec-Fetch-Site': 'same-origin', 'Sec-Fetch-Dest': 'empty', 'Host': '<VCLOUD_URL>', 'Accept-Encoding': 'gzip, deflate, br', 'Sec-Fetch-Mode': 'cors', 'Authorization': '[REDACTED]', 'sec-ch-ua': '"Chromium";v="94", "Google Chrome";v="94", ";Not A Brand";v="99"', 'sec-ch-ua-mobile': '?0', 'sec-ch-ua-platform': '"Windows"', 'Accept-Language': 'en-US,en;q=0.9,fr-FR;q=0.8,fr;q=0.7', 'Content-Length': '580', 'Content-Type': 'application/+json'}, 'body': 'eyJhcGlWZXJzaW9uIjoiY3NlLnZtd2FyZS5jb20vdjIuMCIsImtpbmQiOiJuYXRpdmUiLCJtZXRhZGF0YSI6eyJhZGRpdGlvbmFsUHJvcGVydGllcyI6dHJ1ZSwib3JnTmFtZSI6IkRFTU9fT1JHIiwidmlydHVhbERhdGFDZW50ZXJOYW1lIjoiREVNT19WREMiLCJuYW1lIjoiemRhZHphemRhemRhZHphIiwic2l0ZSI6IiJ9LCJzcGVjIjp7ImFkZGl0aW9uYWxQcm9wZXJ0aWVzIjp0cnVlLCJ0b3BvbG9neSI6eyJjb250cm9sUGxhbmUiOnsiY291bnQiOjEsInNpemluZ0NsYXNzIjoiTCIsInN0b3JhZ2VQcm9maWxlIjoiUkFJRDUifSwid29ya2VycyI6eyJjb3VudCI6Miwic3RvcmFnZVByb2ZpbGUiOiJSQUlENSJ9LCJuZnMiOnsiY291bnQiOjAsInNpemluZ0NsYXNzIjpudWxsLCJzdG9yYWdlUHJvZmlsZSI6bnVsbH19LCJzZXR0aW5ncyI6eyJvdmRjTmV0d29yayI6IkRFTU8tSUFBUy1MQU4iLCJzc2hLZXkiOm51bGwsInJvbGxiYWNrT25GYWlsdXJlIjp0cnVlfSwiZGlzdHJpYnV0aW9uIjp7InRlbXBsYXRlTmFtZSI6InVidW50dS0xNi4wNF9rOC0xLjE4X3dlYXZlLTIuNi41IiwidGVtcGxhdGVSZXZpc2lvbiI6Mn19fQ==', 'statusCode': 0, 'request': True} 21-10-08 14:04:56 | mqtt_publisher:116 - send_response | Request Id: 2e196f03-beec-4be4-ba9b-3a0c93ff5d10 | DEBUG :: publish return (rc, msg_id): (0, 15) 21-10-08 14:04:56 | mqtt_consumer:85 - process_mqtt_message | Request Id: 2e196f03-beec-4be4-ba9b-3a0c93ff5d10 | DEBUG :: MQTT response: {'type': 'API_RESPONSE', 'headers': {'requestId': '2e196f03-beec-4be4-ba9b-3a0c93ff5d10'}, 'httpResponse': {'statusCode': 500, 'headers': {'Content-Type': 'application/json', 'Content-Length': 128}, 'body': 'eyJtZXNzYWdlIjogeyJtaW5vciBlcnJvciBjb2RlIjogLTEsICJlcnJvciBkZXNjcmlwdGlvbiI6ICJbIGM0NTU1MWJiLTVkZDItNDk2Yi1hMjA1LTRlYTZkMGIxZjlhNyBdIFRoaXMgb3BlcmF0aW9uIGlzIGRlbmllZC4ifX0='}}

Thanks for your feedback

[sahithi]

Looks like user doesn't have enough rights. Please publish the "cse:nativeCluster entitlement" right bundle to the tenant org and assign at least "cse:native cluster EDIT right" to the tenant user. And then reattempt the cluster creation

[CrazyVolnay]

It's already setup as advised. The Right Bundle is published to our demo tenant :

In the Demo tenant, a role is setup to provide such rights :

And the role is assigned to the user :

But the user can't deploy a kubernetes cluster :

[sakthisunda]

Thanks for using CSE. Can you please check if these steps work:

1. Please clone vApp Author/Org admin role from tenant (/tenant/DEMO_ORG) portal. (you have used provider portal to create global role as in your screenshot).
2. Add extra rights(CSE:NATIVECLUSTER etc) that are required as you did to this role.
3. Create a user in the demo org and assign the role created in step-2 to this user.
4. Login as the tenant user created in step-3 and then try creating the cluster.
5. If possible, start with clean log file by stop server, clean log , start server.
6. This will create log files fresh. If this fails, we will require log files uploaded (cse-server-debug.log, cloud-api-wire.log)

[goelaashima]

@CrazyVolnay

Let us know if the steps above resolved the issue for you.

Aashima

[CrazyVolnay]

Hi,

I tried to follow advised steps loggued as tenant admin account, but while cloning the VAPP Author role, I cannot see cse rights :

As you can see below, the CSE bundle right is properly published to at least my demo tenant :

The demo org tenant admin has the default Organization Administrator role :

May the tenant admin have more specific rights to see cse rights ?

[sakthisunda]

@CrazyVolnay

Tenant admin with published cse:nativeCluster rights should be able to see the rights when they clone and edit the rights

To reproduce, I followed the steps you tried as follows:

1. Login as sys admin: I created an org DEMO_ORG with an user demoadmin (org administrator)
2. I published the rights: cse:nativeCluster Entitlement to DEMO_ORG
3. Login as tenant admin: demoadmin.(https://vmc-vcloud-dhcp-168-149.eng.vmware.com/tenant). Refer my screenshots for ensuring the login as demoadmin.
4. Cloned the vapp Author role: Please check the browser address: https://<host_url>/tenant/DEMO_ORG/administration/access-control/roles
5. Modified selected rights : I could see CSE native Cluster rights under the section: OTHER
6. I am able to see those rights

My guess is that the persona who logged into may be someone:

• who is org-admin who did not get the cse:Native cluster bundle rights.
• org-admin who is not an user of DEMO_ORG

Screenshots:

Verified the login is demoadmin(org administrator)

Let me know, if this helps.