eclipse-ee4j
Default GF trust store setup causes hard to debug certification failure
The default GF trust store set up leads to a 'javax.net.ssl.SSLHandshakeException with message PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target' on any attempt to access a URL with https. Technically this is correct as set up but a better set up that uses the default Java trust store would avoid this.
Environment Details
Glassfish 7.0.6 java version "17.0.7" 2023-04-18 LTS Java(TM) SE Runtime Environment (build 17.0.7+8-LTS-224) Java HotSpot(TM) 64-Bit Server VM (build 17.0.7+8-LTS-224, mixed mode, sharing) Mac OS 12.4 (Monterey) Database not applicable
Problem Description
By default GF creates a keystore (keystore.jks) and a trust store (cacerts.jks) for each domain. The keystore is populated with two certificates - glassfish-instance and s1as. These two certificates are copied into the trust store. And that is it. Any certificate received in an SSL exchange will fail to be trusted as described for lack of CA certificates in the trust store.
A stack trace is attached.
This situation seems unnecessary because Java provides a trust store populated with all sorts of CA and ICA certificates and most received certificates would end up being trusted as intended.
The problem is reliably reproducible for repeated access to one site or to different sites.
A better set up would be for GF to default to the Java trust store.
An even better set up would be for GF to initially look for a trust path in its own trust store and then, on failure, in the Java trust store. This would allow specific certificates to be trusted by specific domains (by adding them to the domain trust store) without them being trusted by all Java applications on the machine.
This issue matters because the 'unable to find valid certification path' is hard to debug.
I had to:
- do some research to understand that the message is telling me that a received SSL certificate can't be trusted for want of a CA or ICA certificate in the trust store
- some further digging to find that GF uses its own domain-specific trust stores
- some further digging to find out how to get some debugging information on the failure
- extract the identity of the certificates in the received certificates trust chain from the debugging information
- search for the CA certificate on the web, download it, and add it to the domain trust store
only to find that the certificate was already in the Java trust store.
The user work around is easy:
- I change the GF JVM option to point to the Java trust store
- and, if glassfish-instance and s1as do actually need to be trusted, copy them into the Java trust store but the issue could be avoided in the first place.
If, for some reason, GF can't, or shouldn't, use the Java trust store as its default trust store then could we have the above, rather cryptic, error message amplified along the following lines:
unable to find valid certification path to requested target (<URL being accessed>) because none of the following certificates (
This leads directly to resolution. In fact, why don't we amplify the message anyway.
The URL being accessed and the names of the certificates have to be known to GF so adding them to the message isn't burdensome.
Steps to reproduce
- Compile as a web application, deploy, and run the attached GFTrustStoreReproducer against any convenient running glassfish 7 instance
No library JARs are needed; no modular Java --adds are required as JVM options, at least to the point of failure.
Note that the failure may leave the glassfish instance unresponsive.
Impact of Issue
The impact was the time involved to resolve an issue that was inevitable as set up but which could have been avoided with a better set up or whose resolution could have been much simplified with a more informative message.
Hi,
Thanks for the very detailed support. To be honest, I'm not exactly sure what the original rationale was for just having those certificates in the truststore. The error message could be updated regardless, as I agree, it's not super clear what's exactly going on.
To be honest, I'm not exactly sure what the original rationale was for just having those certificates in the truststore.
I'm not sure either but from my past experience, certificates often get expired and have to be renewed, it's a pain to maintain fresh certificates in the default GlassFish installation.
Maybe a better solution would be to import all the certificates from the JDK's cacerts file at first start of the server?
G’day,
… or, as I suggested, do the trust store lookup against the glassfish trust store first and, if that fails, try it again against the JDK’s trust store. Best of both worlds - benefits of the CAs in the JDK store, no certificate copy at start up, no cert maintenance in the default GF installation, and the possibly of glassfish (or domain) specific trusted certificates.
On 31 Jul 2023, at 08:07, Ondro Mihályi @.***> wrote:
To be honest, I'm not exactly sure what the original rationale was for just having those certificates in the truststore.
I'm not sure either but from my past experience, certificates often get expired and have to be renewed, it's a pain to maintain fresh certificates in the default GlassFish installation.
Maybe a better solution would be to import all the certificates from the JDK's cacerts file at first start of the server?
— Reply to this email directly, view it on GitHub https://github.com/eclipse-ee4j/glassfish/issues/24523#issuecomment-1657276941, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKL4ELWCQVDWCRXVVCW23DXS3LJRANCNFSM6AAAAAA24EAKQI. You are receiving this because you authored the thread.
Hi, @Blavo, I suggested copying the certificates from the JDK because I thought it's easier to implement. If there's a single trust store, the SSL context doesn't have to be changed. But, looking into the code, maybe it's enough to modify SecuritySupportImpl.java to add the behavior you suggest.
G’day,
You can safely load the JDK store once for the glassfish session since we won’t make any updates to it from glassfish so don’t need to synchronise with other accesses. Alternatively you can open the JDK store each time you need to access it. That way you don’t need to cycle glassfish to bring in a new or updated certificate. Inevitably we will discover certificate expiry or non-existence mid-session and it would be nice to be able to fix the issue on the fly.
On 31 Jul 2023, at 15:33, Ondro Mihályi @.***> wrote:
Hi, @Blavo https://github.com/Blavo, I suggested copying the certificates from the JDK because I thought it's easier to implement. If there's a single trust store, the SSL context doesn't have to be changed. But, looking into the code, maybe it's enough to modify SecuritySupportImpl.java to add the behavior you suggest.
— Reply to this email directly, view it on GitHub https://github.com/eclipse-ee4j/glassfish/issues/24523#issuecomment-1657651280, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKL4ENCLPG5I7T2Y5NW4J3XS47SLANCNFSM6AAAAAA24EAKQI. You are receiving this because you were mentioned.
G’day, … or, as I suggested, do the trust store lookup against the glassfish trust store first and, if that fails, try it again against the JDK’s trust store. Best of both worlds - benefits of the CAs in the JDK store, no certificate copy at start up, no cert maintenance in the default GF installation, and the possibly of glassfish (or domain) specific trusted certificates. … On 31 Jul 2023, at 08:07, Ondro Mihályi @.***> wrote: To be honest, I'm not exactly sure what the original rationale was for just having those certificates in the truststore. I'm not sure either but from my past experience, certificates often get expired and have to be renewed, it's a pain to maintain fresh certificates in the default GlassFish installation. Maybe a better solution would be to import all the certificates from the JDK's cacerts file at first start of the server? — Reply to this email directly, view it on GitHub <#24523 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKL4ELWCQVDWCRXVVCW23DXS3LJRANCNFSM6AAAAAA24EAKQI. You are receiving this because you authored the thread.
I believe there should be just one cacerts keystore used and properly configured (JVM option) and maintained. We are talking about security, spreading responsibility between several cacerts sounds like a bad idea to me. Also in one repository can be whole chain of certificates. If someone wants to use HTTPS/TLS protocols on production, he should replace that generated self signed certificate by another, signed by some authority. Perhaps we could simplify that in UI and asadmin commands, including importing and maintenance of the repository.
Note yet one thing - in cluster are these two repositories replicated, while you cannot replicate keystores of the operating system and JDK.
Also I am not sure how it would work with the CRL.
G’day,
I accept that there are other considerations to the two-keystone mechanism I suggested. I would be happy just to expand the error message along the lines that I suggested. Then the issue is clear and the user can decide how best to proceed in their circumstances.
Looking at the issue in GitHub I see some parts of the message text I suggested were corrupted because I used angle brackets to indicate variables. So I’ve restated an expanded version of the suggested message text below:
"unable to find valid certification path to requested target (#URL#) because none of the following certificates (#leaf cert name#, #ICA cert name#, #CA cert name#) were found in the key store indicated by the glassfish trust store VM option at #path to GF trust store#; one or more of these certificates may be available in the JDK trust store at #path to JDK trust store#."
All of the necessary information to make the message should be to hand in the code since it would have been needed to get to the point of failure. The message is verbose but gives the user all the information necessary to fix the issue by one of several paths.
On 31 Jul 2023, at 16:12, David Matějček @.***> wrote:
G’day, … or, as I suggested, do the trust store lookup against the glassfish trust store first and, if that fails, try it again against the JDK’s trust store. Best of both worlds - benefits of the CAs in the JDK store, no certificate copy at start up, no cert maintenance in the default GF installation, and the possibly of glassfish (or domain) specific trusted certificates. … x-msg://15/# On 31 Jul 2023, at 08:07, Ondro Mihályi @.***> wrote: To be honest, I'm not exactly sure what the original rationale was for just having those certificates in the truststore. I'm not sure either but from my past experience, certificates often get expired and have to be renewed, it's a pain to maintain fresh certificates in the default GlassFish installation. Maybe a better solution would be to import all the certificates from the JDK's cacerts file at first start of the server? — Reply to this email directly, view it on GitHub <#24523 (comment) https://github.com/eclipse-ee4j/glassfish/issues/24523#issuecomment-1657276941>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKL4ELWCQVDWCRXVVCW23DXS3LJRANCNFSM6AAAAAA24EAKQI https://github.com/notifications/unsubscribe-auth/AOKL4ELWCQVDWCRXVVCW23DXS3LJRANCNFSM6AAAAAA24EAKQI. You are receiving this because you authored the thread.
I believe there should be just one cacerts keystore used and properly configured (JVM option) and maintained. We are talking about security, spreading responsibility between several cacerts sounds like a bad idea to me. Also in one repository can be whole chain of certificates. If someone wants to use HTTPS/TLS protocols on production, he should replace that generated self signed certificate by another, signed by some authority. Perhaps we could simplify that in UI and asadmin commands, including importing and maintenance of the repository.
Not yet one thing - in cluster are these two repositories replicated, while you cannot replicate keystores of the operating system and JDK.
Also I am not sure how it would work with the CRL.
— Reply to this email directly, view it on GitHub https://github.com/eclipse-ee4j/glassfish/issues/24523#issuecomment-1657728080, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKL4EIMD2SPVOXHMYPA6QLXS5EEVANCNFSM6AAAAAA24EAKQI. You are receiving this because you were mentioned.
I think GlassFish, which also runs as a server, should keep its truststore to a minimum. If we really want to, we should do the following at our own risk before launching GlassFish:
keytool -importkeystore -v -noprompt -srckeystore "$JAVA_HOME/jre/lib/security/cacerts" -destkeystore "$GLASSFISH_HOME/glassfish/domains/domain1/config/cacerts.jks" -deststoretype jks
It may be difficult to get the message out on GlassFish because GlassFish cannot intercept JDK method calls made by user apps.
Unable to make request because of javax.net.ssl.SSLHandshakeException with message PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target|#]
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:578)
at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:123)
at gftruststorereproducer.GFTrustStoreReproducer.initialise(GFTrustStoreReproducer.java:28)
G'day,
Looking at the stack trace I see org.glassfish.ejb.startup…. mentioned. We could trap THIS error there. But that doesn’t help in the general case where a trust store check could be invoked by the user in all sorts of ways.
The exception appears to come from with a security provider. That maybe provides an opportunity but I doubt the issue is worth the effort.
We appear to be in a bind: we aren’t willing (on reasonable grounds) to prevent the error happening; and we can’t intervene to help the user when the error (inevitably) occurs.
On 1 Aug 2023, at 15:45, Hiroki Sawamura @.***> wrote:
I think GlassFish, which also runs as a server, should keep its truststore to a minimum. If we really want to, we should do the following at our own risk before launching GlassFish:
keytool -importkeystore -v -noprompt -srckeystore "$JAVA_HOME/jre/lib/security/cacerts" -destkeystore "$GLASSFISH_HOME/glassfish/domains/domain1/config/cacerts.jks" -deststoretype jks It may be difficult to get the message out on GlassFish because GlassFish cannot intercept JDK method calls made by user apps.
Unable to make request because of javax.net.ssl.SSLHandshakeException with message PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target|#] javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:578) at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:123) at gftruststorereproducer.GFTrustStoreReproducer.initialise(GFTrustStoreReproducer.java:28) — Reply to this email directly, view it on GitHub https://github.com/eclipse-ee4j/glassfish/issues/24523#issuecomment-1659605143, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKL4EOY6JHI47YQROZHHC3XTCJYVANCNFSM6AAAAAA24EAKQI. You are receiving this because you were mentioned.
This issue has been marked as inactive and old and will be closed in 7 days if there is no further activity. If you want the issue to remain open please add a comment
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@hs536,
Would it be a good solution to add the following to GlassFish?
- an asadmin command, that would allow adding all certificates from the JDK truststore, or resetting the GF truststore to the default set of certificates
- add an option to the create-domain command to copy certificates from the JDK truststore to the GF truststore
G’day,
Yes, I think these would be useful in general. Presumably these would result in modification to the domain.xml file so that the command doesn’t have to be re-entered on each start up.
But this won’t help those that use embedded glassfish (like me). For such users an option in the domain.xml file to copy all certificates in the JDK trust store to the GF trust store would work. You could generalise the option so that it takes the path to a trust store as a parameter, so the certificates can be copied from an arbitrary location. The benefit is that addition of trusted certificates can be done on a per-domain basis rather than being global to all domains. However this has obvious security implications since it would be easier to slip in an unauthorised root certificate.
Ian Blavins 0408 848 548
On 7 Feb 2025, at 00:49, Ondro Mihályi @.***> wrote:
@hs536 https://github.com/hs536,
Would it be a good solution to add the following to GlassFish?
an asadmin command, that would allow adding all certificates from the JDK truststore, or resetting the GF truststore to the default set of certificates add an option to the create-domain command to copy certificates from the JDK truststore to the GF truststore — Reply to this email directly, view it on GitHub https://github.com/eclipse-ee4j/glassfish/issues/24523#issuecomment-2639881544, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKL4EL4Y6IMYLI5V6LJQ532ONR5FAVCNFSM6AAAAABLZPCTJCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMZZHA4DCNJUGQ. You are receiving this because you were mentioned.
I did some quick experiments and I think it's all possible. Here are things to consider:
- There's no way to find out the path to the default truststore in the JDK in a standard way. However, it's possible to load the default truststore by standard Java API (Java 11+) if the
javax.net.ssl.trustStoresystem property is not set - GlassFish, in default configuration, sets the
javax.net.ssl.trustStoreas a JVM option. With this, the default truststore file is the one inf the GlassFish domain - After JVM starts, it's possible to set or unset the
javax.net.ssl.trustStoreproperty with immediate impact. So it's possible to unset that property after JVM starts, load the default JDK trust store, and then set it back to the original value. I'm not sure if the new certificates would be picked up without a restart - I don't know when and how often JVM or GlassFish look into the truststore. If they just do it once after startup, that might be an issue
An asadmin command could access the default JDK truststore even without knowing the path to it, if it temporarily unsets the javax.net.ssl.trustStore property. It could copy all the certificates to the GlassFish truststore from the default truststore or from a given file. We could also add an option to run it on startup if needed. The same asadmin command could remove expired certificates.
In GlassFish Embedded, it's also possible to launch the same asadmin commands, so it shouldn't be an issue to start GlassFish Embedded and immediately run the asadmin command from the JVM. The runnable JAR also supports running asadmin commands, either specified as a command line argument or in a property file.
G’day,
Re the embedded glassfish option: if, rather than running an asadmin command after embedded glassfish has started, we instead use an entry in the domain.xml file then the certificates can be loaded to the glassfish trust store before embedded glassfish starts, avoiding any issue of maybe having to restart glassfish after the asadmin command. With an entry in the domain.xml file the certificate copying can be done immediately after the glassfish trust store has been set up; If a user intends to run glassfish embedded then they have to provide a domain.xml file so adding a trusted certificate related entry to the file is no burden.
In fact, if you implement an asadmin option for non-embedded glassfish, then the domain.xml file has to be modified when the asadmin command runs so that the option will be honoured on each subsequent glassfish start without redoing the asadmin command. So there pretty much has to be an entry in the domain xml file and ideally embedded glassfish should deal with it in the same way as non-embedded glassfish. So no need for the embedded glassfish user to run an asadmin command - they just put the appropriate entry in the domain.xml file.
Ian Blavins 0408 848 548
On 14 Feb 2025, at 21:58, Ondro Mihályi @.*** @.***>> wrote:
OndroMih left a comment (eclipse-ee4j/glassfish#24523) I did some quick experiments and I think it's all possible. Here are things to consider:
There's no way to find out the path to the default truststore in the JDK in a standard way. However, it's possible to load the default truststore by standard Java API (Java 11+) if the javax.net.ssl.trustStore system property is not set GlassFish, in default configuration, sets the javax.net.ssl.trustStore as a JVM option. With this, the default truststore file is the one inf the GlassFish domain After JVM starts, it's possible to set or unset the javax.net.ssl.trustStore property with immediate impact. So it's possible to unset that property after JVM starts, load the default JDK trust store, and then set it back to the original value. I'm not sure if the new certificates would be picked up without a restart - I don't know when and how often JVM or GlassFish look into the truststore. If they just do it once after startup, that might be an issue An asadmin command could access the default JDK truststore even without knowing the path to it, if it temporarily unsets the javax.net.ssl.trustStore property. It could copy all the certificates to the GlassFish truststore from the default truststore or from a given file. We could also add an option to run it on startup if needed. The same asadmin command could remove expired certificates.
In GlassFish Embedded, it's also possible to launch the same asadmin commands, so it shouldn't be an issue to start GlassFish Embedded and immediately run the asadmin command from the JVM. The runnable JAR also supports running asadmin commands, either specified as a command line argument or in a property file.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
https://github.com/eclipse-ee4j/glassfish/issues/24523#issuecomment-2658985242 https://github.com/notifications/unsubscribe-auth/AOKL4EKJNKYEEEVIYTWKWKT2PXD5TAVCNFSM6AAAAABLZPCTJCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJYHE4DKMRUGI
OndroMih left a comment (eclipse-ee4j/glassfish#24523) https://github.com/eclipse-ee4j/glassfish/issues/24523#issuecomment-2658985242 I did some quick experiments and I think it's all possible. Here are things to consider:
There's no way to find out the path to the default truststore in the JDK in a standard way. However, it's possible to load the default truststore by standard Java API (Java 11+) if the javax.net.ssl.trustStore system property is not set GlassFish, in default configuration, sets the javax.net.ssl.trustStore as a JVM option. With this, the default truststore file is the one inf the GlassFish domain After JVM starts, it's possible to set or unset the javax.net.ssl.trustStore property with immediate impact. So it's possible to unset that property after JVM starts, load the default JDK trust store, and then set it back to the original value. I'm not sure if the new certificates would be picked up without a restart - I don't know when and how often JVM or GlassFish look into the truststore. If they just do it once after startup, that might be an issue An asadmin command could access the default JDK truststore even without knowing the path to it, if it temporarily unsets the javax.net.ssl.trustStore property. It could copy all the certificates to the GlassFish truststore from the default truststore or from a given file. We could also add an option to run it on startup if needed. The same asadmin command could remove expired certificates.
In GlassFish Embedded, it's also possible to launch the same asadmin commands, so it shouldn't be an issue to start GlassFish Embedded and immediately run the asadmin command from the JVM. The runnable JAR also supports running asadmin commands, either specified as a command line argument or in a property file.
— Reply to this email directly, view it on GitHub https://github.com/eclipse-ee4j/glassfish/issues/24523#issuecomment-2658985242, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKL4EKJNKYEEEVIYTWKWKT2PXD5TAVCNFSM6AAAAABLZPCTJCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJYHE4DKMRUGI. You are receiving this because you were mentioned.
In glassfish 5 there were a lot of important certificates already provided. Why was it decided to keep them now to a minimum?
I believe this is already resolved in 7.1.0 ... defaults are gone from embedded. They even made no sense.