ZWEAM511E causes ZWES1605W (JWT endpoint issue)

[ActionThisDay]

Im doing a new V2 install , and seeing some issues , possibly related. The main one being I fail authentication to the ZOWE Explorer

I cant login to ZOWE Explorer - I get an error Authentication Failed for 3 Types [saf,apiml,zss] APIML 401

My Server sysout reports 2022-10-11 11:26:50.741 ZWED:424 ZWESVUSR WARN (_zsf.auth,webauth.js:378) ZWED0003W - User=undefined (org.zowe.zlux.auth.safsso): (org.zowe.zlux.auth.safsso): Session authenticate failed. Plugin response: {"success":false,"reason":"Unknown","error":{"message":" APIML 401 "},"apiml":true,"zss":true,"sso":true,"canChangePassword":true} 2022-10-11 11:26:50.745 ZWED:424 ZWESVUSR INFO (_zsf.auth,webauth.js:375) ZWED0070I - User=undefined (org.zowe.zlux.auth.trivial) (org.zowe.zlux.auth.trivial): Session authenticate successful. Plugin response: {"success":true}

ZIS status is good ZWES1014I ZIS status - 'Ok' (name='ZWESIS_STD ', cmsRC='0', description='Ok', clientVersion='2')

The instance settings ZWE_zowe_setup_security_product=RACF (? should that be SAF ???) ZWE_zowe_setup_security_groups_admin=ZWEADMIN ZWE_zowe_setup_security_groups_stc=ZWEADMIN ZWE_zowe_setup_security_groups_sysProg=ZWEADMIN ZWE_zowe_setup_security_users_zowe=ZWESVUSR ZWE_zowe_setup_security_users_zis=ZWESIUSR ZWE_zowe_setup_security_stcs_zowe=ZWESLSTC (? is this okay for a zwe start from OMVS ???) ZWE_zowe_setup_security_stcs_zis=ZWESISTC ZWE_zowe_setup_security_stcs_aux=ZWESASTC

I use OMVS 'zwe start' to bring up the server , note the jobname is not ZWESLSTC. JOBNAME StepName ProcStep ZWE1SV ZWE1SV ZWELNCH ZWESISTC ZWESISTC ZWESIS01

The ZSSserver also streams ZWES1606W Failed to get JWK - HTTP response error, retry in 10 seconds ZWES1606W Failed to get JWK - HTTP response error, retry in 10 seconds ZWES1605W Server will not accept JWT

Appreciate any guidance to direct me where the issue is or how to debug further.

[JoeNemo]

Please turn on debugging and repeat. Zowe 2.4 and later will log JWK issues by default.

[ActionThisDay]

Thank you for reply Joe.. For the APIML 401 Login failure I did not see anything in the log 2022-10-12 16:30:17 ZWELNCH:33554601 ZWESVUSR DEBUG waitpid RC = 0 for files-api(900) 2022-10-12 16:30:17 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from discovery(819) 2022-10-12 16:30:17 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from app-server(67109058) 2022-10-12 16:30:17 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from gateway(16777520) 2022-10-12 16:30:17 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from zss(67109561) 2022-10-12 16:30:17 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from jobs-api(850) 2022-10-12 16:30:17 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from metrics-service(33555117) 2022-10-12 16:30:17 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from api-catalog(17) 2022-10-12 16:30:17.041 ZWED:376 ZWESVUSR WARN (_zsf.auth,webauth.js:378) ZWED0003W - User=undefined (org.zowe.zlux.auth.safsso): 2022-10-12 16:30:17.052 ZWED:376 ZWESVUSR INFO (_zsf.auth,webauth.js:375) ZWED0070I - User=undefined (org.zowe.zlux.auth.trivial) 2022-10-12 16:30:18 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from files-api(900) 2022-10-12 16:30:18 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from caching-service(16777245) 2022-10-12 16:30:18 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from discovery(819) 2022-10-12 16:30:18 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from gateway(16777520) 2022-10-12 16:30:18 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from jobs-api(850) 2022-10-12 16:30:18 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from zss(67109561) 2022-10-12 16:30:18 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from metrics-service(33555117) 2022-10-12 16:30:18 ZWELNCH:33554601 ZWESVUSR DEBUG waitpid RC = 0 for discovery(819)

Whilst the 'Failed to get JWK' did not have much additional info either with DEBUG ON ZWES1600I JWT will be configured using JWK URL https://abc.def.ghi.com:7554/gateway/api/v1/auth/keys/public/current ZWES1039I Installing 'VSAM dataset contents' service... ZWES1039I Installing 'dataset metadata' service... ZWES1039I Installing 'dataset contents' service... ZWES1606W Failed to get JWK - failed to send HTTP request, retry in 10 seconds ZWES1014I ZIS status - 'Ok' (name='ZWESIS_STD ', cmsRC='0', description='Ok', clientVersion='2') 2022-10-12 15:22:37 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from api-catalog(17) 2022-10-12 15:22:37 ZWELNCH:33554601 ZWESVUSR DEBUG waiting for next message from files-api(900)

If I try this URL direct . {"messages":[{"messageType":"ERROR","messageNumber":"ZWEAM511E","messageContent":"There was a TLS request error accessing the URL '/gateway/api/v1/auth/keys/public/current': 'Unsupported or unrecognized SSL message'","messageAction":"Possible actions regarding to message content:\n - Message: The certificate is not trusted by the API Gateway.\n Action: Verify trust of the certificate is the issue by disabling certificate verification and retry the request.\n - Message: Certificate does not match any of the subject alternative names.\n Action: Verify that the hostname which the certificate is issued for matches the hostname of the service.\n - Message: Unable to find the valid certification path to the requested target.\n Action: Import the root CA that issued services' certificate to API Gateway truststore.\n - Message: Verify the requested service supports TLS.\n Action: Ensure the requested service is running with TLS enabled.\n - Message: Review the APIML debug log for more information.\n Action: Enable APIML debug mode and retry the request, then review the APIML log for TLS errors.","messageReason":"The Gateway refuses to communicate with the requested service.","messageKey":"org.zowe.apiml.common.tlsError"}]}

[1000TurquoisePogs]

Hi all, I transfered this upon @ActionThisDay 's insight that the gateway endpoint could have been the cause. Perhaps the advice in https://docs.zowe.org/v1.28.x/troubleshoot/troubleshoot-apiml-error-codes/#zweam511e will solve this.

[achmelo]

@ActionThisDay the response you are receiving when you call URL directly looks like you are sending a client certificate in the request, please verify it is not the case. I believe this is not the same problem as you have in ZSS. There is no reason for ZSS to use a client certificate for this request. To fix your ZSS JWK issue, please make sure that ZSS has CA certificate configured to trust API ML and API ML certificate(which I think is the same for every zowe component) has the correct hostname among SAN. Also, verify that API ML hostname can be resolved by ZSS.

[anton-brezina]

@ActionThisDay is the issue still valid or did you had a chance to try the suggestions above?

[NayerNajafi]

@anton-brezina Hello, I have the same issue with Zowe 2.4. The repeated message ZWES1606W appears in Zowe job log when I start Zowe. I am using signed trusted certificate foe all Zowe components. Also, I am able to access Desktop and explorers, API-Catalog. ZWES1606W Failed to get JWK - failed to send HTTP request, retry in 10 seconds HTTP status 404
failed to obtain JWK, status = 2

[achmelo]

@NayerNajafi could you please verify that you can access Gateway service at https://<host>:<gateway-port>/gateway/api/v1/auth/keys/public/current , this is the URL that ZSS calls to retrieve JWK

[NayerNajafi]

@achmelo I tried above URL and got the following error: {"messages":[{"messageType":"ERROR","messageNumber":"ZWEAM511E","messageContent":"There was a TLS request error accessing the URL '/gateway/api/v1/auth/keys/public/current': 'Unsupported or unrecognized SSL message'","messageAction":"Possible actions regarding to message content:\n - Message: The certificate is not trusted by the API Gateway.\n Action: Verify trust of the certificate is the issue by disabling certificate verification and retry the request.\n - Message: Certificate does not match any of the subject alternative names.\n Action: Verify that the hostname which the certificate is issued for matches the hostname of the service.\n - Message: Unable to find the valid certification path to the requested target.\n Action: Import the root CA that issued services' certificate to API Gateway truststore.\n - Message: Verify the requested service supports TLS.\n Action: Ensure the requested service is running with TLS enabled.\n - Message: Review the APIML debug log for more information.\n Action: Enable APIML debug mode and retry the request, then review the APIML log for TLS errors.","messageReason":"The Gateway refuses to communicate with the requested service.","messageKey":"org.zowe.apiml.common.tlsError"}]}

[achmelo]

@NayerNajafi do you have correct hostname among SAN names in certificate?

[NayerNajafi]

I didn't. I just specified SAN info in certificate. Do I need to re-run zwe init certificate after this change?

[achmelo]

I don't know what is your setup. What do you mean by I just specified SAN info in certificate. ? Did you generate certificates using zowe scripts?

[NayerNajafi]

Sorry, forget my last comment. Yes, I have the correct hostname among SAN in my certificate

[achmelo]

another problem could be zOSMF certificate. Is the certificate authority of zOSMF imported in the Zowe truststore/keyring? Does the certificate have a correct hostname among SAN names?

[balhar-jakub]

There are no further comments, and as such we are closing the issue. Feel free to reopen the issue of there is still something to help with.