zowe
Gateway cannot "ignore" client certificates unrelated to Zowe use.
I've heard for the second time that users are having issues with client certificate usage when using the gateway.
Basically, if a user has client certificates in their organization, but the certificates have nothing to do with a zowe account, then it's possible that going to a website behind the gateway will have the browser prompt the user to choose one of the certificates. Clicking on any one will result in rejection, instead when prompted by the browser, the user must select "cancel" to not send any client certificate since none are applicable.
This prompt happens to them daily, and even if a few people in the company know to always hit "cancel", it's a "bug" generator in that every other user will just complain Zowe is broken, or at least is very annoying - like a GDPR cookie popup that also breaks a site if you click the wrong button.
In this case, the user is not able to either remove all client certs, or have all client certs related to a Zowe user.
The source of the problem appears to be that Zowe uses client certificate verification between Zowe's servers, but that when doing so, it also involves all clients such as browsers?
Potential solution If the browser's response to show a client certificate prompt is due to an HTTP header sent by the gateway, then why not have an admin-configurable exclusion list of clients, or an inclusion list of servers, for which to not send the header that triggers the behavior. Then, the servers could continue to use client certs to each other, without these users being impacted by a broken prompt every time they open their browser to Zowe.
That's a good point. We saw the same behavior at least with one user as well.
Okay, when we met we said we'd split this into two investigations: what can java / our stack do at the TLS level, and what can AT-TLS do
It appears AT-TLS can do that proposal of having client cert requests be conditional to a range of IPs.
My understanding is:
-
A TTLSRule defines an IP & Port range for which some behavior should apply, defined in TTLSEnvironmentAction / TTLSConnectionAction but pointed to by TTLSRule
-
TTLSEnvironmentAction / TTLSConnectionAction both have a parameter HandshakeRole which can be Client / Server / ServerWithClientAuth. TTLSConnectionAction overrides TTLSEnvironmentAction for this parameter.
-
ServerWithClientAuth = present your cert, and request client servers.
-
Server = present your cert, but dont request client certs. We see an old example of how these chain together here https://docs.zowe.org/stable/user-guide/mvd-configuration/#defining-the-at-tls-rule
-
TTLSEnvironmentAdvancedParms extends ServerWithClientAuth with a few behaviors, ClientAuthType=PassThru, Full, Required
-
PassThru = client cert request disabled
-
Full = client cert requested, but not demanded
-
Required = client cert requested & demanded.
All the statements possible for AT-TLS config datasets, and their params are here https://www.ibm.com/docs/en/zos/2.4.0?topic=applications-tls-policy-statements
It seems to me then, that by having 2 of each of the above objects, you could have different behaviors for different IPs.
For example,
TTLSRule ATTLS1~GW
{
LocalAddr All
RemoteAddr 127.0.0.1
LocalPortRange [gw_port]
Priority 255
Direction Inbound
...
TTLSEnvironmentActionRef eAct1~GW
}
TTLSEnvironmentAction eAct1~GW
{
HandshakeRole ServerWithClientAuth
TTLSEnvironmentAdvancedParmsRef eAdv1~GW
...
}
TTLSEnvironmentAdvancedParms eAdv1~GW
{
ClientAuthType Full
}
Note priority here. 255 for the above, 254 for below, for handling the IP ranges.
TTLSRule ATTLS2~GW
{
LocalAddr All
RemoteAddr All
LocalPortRange [gw_port]
Priority 254
Direction Inbound
...
TTLSConnectionActionRef cAct2~GW
}
TTLSConnectionAction cAct2~GW
{
HandshakeRole Server
...
}
Finally, since gateway still needs to contact others, I believe Direction Outbound is needed, in a third rule, for HandshakeRole Client?
TTLSRule ATTLS3~GW
{
LocalAddr All
RemoteAddr All
LocalPortRange [gw_port]
Priority 255
Direction Outbound
...
TTLSConnectionActionRef cAct3~GW
}
TTLSConnectionAction cAct3~GW
{
HandshakeRole Client
...
}
I have yet to test this.
As I heard your squad has been working with AT-TLS recently, how do these rules match up with your understanding? What requirements do you have in the gateway to disable HTTPS so that AT-TLS can be used on top? Perhaps I could test this.
With the TLS there is no way how to process an unknown certificate. The treatment is done before our code and it is just possible to control the process:
- use a different mode to treat (see
server.ssl.client-auth)- NONE - certificates are not supported (one without any choosing)
- WANT - certificates are supported, but not mandatory (used by GW)
- NEED - certificate is required
- add into the trust store the certificate to accept it
We cannot change the logic, once certificates are supported it asks during the handshake for the certificate and if the certificate is not trusted (against the trust store) the request is rejected. If the solution cannot be to disable the client certificate or add the certificate into the trust store, there is just one potential solution to use a different domain to avoid matching the certificate (and could not be offered).
As you mentioned above with AT-TLS it is possible to define different rules (with different ClientAuthType values). It could make a difference and by using priority change the behaviour. APIML takes the certificate from AT-TLS and because of transparency, APIML cannot influence the behavior itself.
Just keep in mind that the specific rule could be controlled just by remote IP which means the IP could be also a firewall, not the user itself.
It makes more sense to me to publish multiple ports with a different setup. It is possible to use an internal port of gateway, see:
server.internal.enabled: true
server.internal.port: <the second port without client certificates>
server.internal.ssl.clientAuth: NONE
Anyway, I guess, the proper solution is to configure the web browser, for example, defining multiple profiles (one with certificates and the other without).
Thanks @pj892031 I would also prefer to use AT-TLS here because I don't know what behavior to expect when changing the "internal" port configuration. It's not documented much. It's in the schema but I'm not sure what communication is "internal" such that it would be used by this port rather than the "external" port.
But, the user I'm working with says actually they not only prefer the AT-TLS solution, but they wish all of Zowe could be AT-TLS configured.
So, I'm setting up a test for them to confirm the behavior of AT-TLS.
Is this document https://docs.zowe.org/stable/user-guide/api-mediation/configuration-at-tls/#at-tls-configuration-for-zowe still correct for turning HTTPS off in gateway in order to use AT-TLS?
I am in the process of setting up such an AT-TLS solution to confirm behavior and will report back.
Happy to report that an AT-TLS solution was usable for this purpose.
However, my previous commented Outbound rule was incorrect.
You cannot use
LocalPortRange [gw_port] Direction Outbound
It should have been obvious before but... requests to not originate from the server listening port.
So the rules that we used instead had to do job filters for outbound.
Inbound = filter by port Outbound = filter by jobname
You could filter by jobname for both though. Just keep in mind that our jobnames are scattered. In v2.17.0 it seems all of zowe will be unified in attls so a simple jobname filter may just be "ZWE*" But in v2.16.0 and under, when you have to exclude certain servers, you end up with complex filter:
"ZWEA*" for most of api-layer repo Except "ZWEM*" and "ZWEC*" for metrics & caching And "ZWES*" for zss, and "ZWED*" for app-server (though app-server attls is broken in older zowe versions, so it got excluded in this way)