security
security copied to clipboard
opensearch-project
[Feature Request] Support for FIPS 140-2 enforced mode.
In some customer service contracts (especially Government contracts), there is a requirement to run all services in FIPS 140-2 compliant/enforced mode.
Elasticsearch supports this currently in their xpack security plugin.
We, and I imagine others could benefit greatly with the ability to run OpenSearch in a FIPS 140-2 enforced mode JVM.
I see there was also an OpenDistro community post asking for this feature as well.
Thanks for reaching out. I'll move this to our security plugin team.
Also @setiah probably you might interested!
A related discussion around making the crypto provider in security plugin configurable. https://github.com/opensearch-project/OpenSearch/issues/1487 As part of this, it should also enable users to provide a FIPS compliant provider.
Please consider how/whether/when this will work on Windows, which has its own world of cryptographic providers.
@davidlago any timeline/roadmap on FIPS compliance for opensearch on windows? Our customers want our application to be FIPS compliant by mid this year and since we use opensearch we would like to see this soon.
Thanks for the interest, @bhupendra-mskhatri! We are not actively working on this issue at the moment, though it is a contribution we'd most definitely welcome.
Thankyou @davidlago for your response. Can you or someone from your team give a basic idea of what changes might be required in opensearch to make it FIPS compliant. This would help me understand the contribution estimate.
Hi @bhupendra-mskhatri, I tried to gather some information for you to be able to gauge the scope.
For starters, we are going to have to rework the use of Bouncy Castle. If you look at their website you can see that while they do have FIPS certified APIs for Java, they are only certified for 1.7, 1.8, and 1.11. This means that the 2.x line and main which supports 1.17 will have to be run with 1.11 in order to be compliant. This is probably acceptable if a little bit confusing (there will be OpenSearch versions that are only compliant in one version configuration).
You can see the Bouncy Castle FIPS versions here and some implementation details here. In effect, you would minimally need to replace any non-compliant algorithms with their certified counterparts from one of the linked Bouncy Castle API versions and make sure that the software then met the guidelines of the documentation.
Thanks, @scrawfor99. Our application bundles OpenSearch as a repository to persist some related settings. In order to make our application FIPS compliant we are running this whole setup under the FIPS setup.
OpenSearch crashes when it runs with FIPS on the mode by saying -
Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.
at sun.security.util.DerInputStream.getLength(DerInputStream.java:606) ~[?:?]
at sun.security.util.DerValue.init(DerValue.java:390) ~[?:?]
at sun.security.util.DerValue.
** @davidlago @saratvemulapalli, Please if anyone came across with similar situation?**
FIPS settings turned on the windows box - OS settings
- In Control Panel, click Administrative Tools, and then double-click Local Security Policy.
- In Security Settings, expand Local Policies and then click Security Options.
- Under Policy in the right pane, double-click System cryptography: Use FIPS-compliant algorithms for encryption, hashing, and signing, and then click enable.
Java settings Update the precedence of Sun’s crypto modules JCE & JSSE to 1 & 2 under java
jdk-11.0.11.9-hotspot\conf\security\java.security security.provider.1=SunJCE security.provider.2=SunJSSE . . security.provider.3=SUN
@rathiyajiv That error message is based on the certificate being loaded, is it possible the cert isn't in PKCS12 format?
Relevant StackOverflow question/answers https://stackoverflow.com/a/57683701/533057
@rathiyajiv That error message is based on the certificate being loaded, is it possible the cert isn't in PKCS12 format?
Relevant StackOverflow question/answers https://stackoverflow.com/a/57683701/533057
Thanks @peternied, the observation here is an error occurs only when FIPS settings are on. Does PKCS12 format a must for FIPS mode from the opeansearch perspective?
Referencing the bouncy castle guide, PKCS12 is not supported in approved-mode. More details from chapter 7 Key Stores https://downloads.bouncycastle.org/fips-java/BC-FJA-UserGuide-1.0.2.pdf
After switch to a supported keystore this should get you around that error.
Hi @rathiyajiv, thank you for following up. I see Peter already provided some more information to help you along your way, but I wanted to address your question about the format. PKCS12 is not a must that I am aware of. That being said, there are several factors that go into FIPS compliance which I am not an expert on.
You will notice on the intial RFC that the requirements are listed for authentication as:
FIPS does not have any specific guidelines for the type of algorithm used so long as it is compatible with these requirements. Instead you can use
The annex of this draft has a list of some of the approved options for implementation. If there is a modification you wish to make to your local OpenSearch configuration in order to be able to run the enforced mode, I would encourage you to do so.
If you were then able to let us know what you did, we may be able to make it a configurable option on the plugin so that it was easier to set up in the future.
Thanks @peternied , @scrawfor99 -
There are some traces around the FIPS in the code @ below https://github.com/opensearch-project/OpenSearch/tree/main/buildSrc/src/main/resources
All the version of security file fips_java_xx_x.security contains the security.provider.1 which points to the bouncycastle jcajce & jsse modules. While the java's /conf/security/java.security refers to SunJCE & SunJSSE.
Can anyone suggest where to find the fips_java_bcjsse_11.security under Opensearch's deployed folder structure? We are on Opensearch version 2.4.1
I've been having a look at the security plugin, and I think the hard-coded dependency on the BouncyCastle provider will need to be broken for this to work. I think is mentioned on another related ticket about abstracting this particular dependency.
So far from my digging, I see the security plugin uses BouncyCastle directly for:
- Security Provider itself which is hard-coded to be loaded on start-up
- OpenBSD BCrypt for authentication
- Blake2b for data anonymization
- ASN.1 decoding
- Hex and Base64 encoding/decoding
If we can replace these with some non-hardcoded implementations or other libraries. Then it should be possible to try and run OpenSearch with another security provider - like the BouncyCastle FIPS provider.
I've been having a look at the security plugin, and I think the hard-coded dependency on the BouncyCastle provider will need to be broken for this to work. I think is mentioned on another related ticket about abstracting this particular dependency.
So far from my digging, I see the security plugin uses BouncyCastle directly for:
- Security Provider itself which is hard-coded to be loaded on start-up
- OpenBSD BCrypt for authentication
- Blake2b for data anonymization
- ASN.1 decoding
- Hex and Base64 encoding/decoding
If we can replace these with some non-hardcoded implementations or other libraries. Then it should be possible to try and run OpenSearch with another security provider - like the BouncyCastle FIPS provider.
As you said BouncyCastle is hard coded means if we use BouncyCastle provider in JVM then Opensearch will Run in Fips mode automatically as BouncyCastle is by default provider
Hi @iainjacksonsas,
Thanks for the details on this. I know you are taking a further look, but to clarify, the major change to make OpenSearch FIPS compliant is swapping out Bouncy Castle use? If changes were made to make another library an option for running things, would that be enough to be considered compliant? Or would further changes on top of that be required?
Thank you.
Yes, looking at the usage of the BouncyCastle provider is a starting point. The key requirement is to run on a FIPS enforcing platform (e.g. RHEL 8) using FIPS certified cryptography.
The major issue stems from OpenSearch Security plugin's direct use of classes in the BouncyCastle provider, which would need replaced with the FIPS certified version and run on a certified JDK (e.g. Java 11). However you cannot have both the FIPS and non-FIPS BouncyCastle providers on the classpath at the same time as they interfere with one another. You must use one or the other.
I currently have a version of the OpenSearch Security plugin locally which does not have any direct dependencies on BouncyCastle. I did some code surgery to take from BouncyCastle just what was needed. This is not a solution, just working through what changes could and need to be made to make this work.
I am currently working through trying to get it to start-up on a FIPS enforcing platform. Still not there, but i'm getting closer.
Any update on FIPS implementation? I can see when I am using BC provide in java.security as below
security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider C:HYBRID;ENABLE{All}; security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS
opensearch is starting fine . Can we assume its running in FIPS mode.
I'm still pulling together my notes. But to answer your question, no it's not quite running in FIPS mode.
The JVM - particularly on RedHat 8 - behaves differently when the kernel is set to FIPS mode. Also, to be running in a FIPS-compliant mode we can't use certain algorithms (e.g. BCrypt).
I'm hoping to get a ticket raised this week with all the details on my investigation.
Thanks for update. I am using RedHat 8 with kernel is set to FIPS mode true and opensearch is running fine with above mentioned java.security setting.
I'm still pulling together my notes. But to answer your question, no it's not quite running in FIPS mode.
The JVM - particularly on RedHat 8 - behaves differently when the kernel is set to FIPS mode. Also, to be running in a FIPS-compliant mode we can't use certain algorithms (e.g. BCrypt).
I'm hoping to get a ticket raised this week with all the details on my investigation.
Hi @iainjacksonsas Is there a checklist we can go against to verify and conclude that after the changes we make for FIPS compliance and enabling FIPS at OS level, opensearch is fully FIPS compliant or is not?
e.g. to @szwlhd query you responded that though after his changes opensearch started successfully, but it isn't running in FIPS mode. What are those references against which you cross checked to identify whether opensearch is running in FIPS mode or not.
There isn't a checklist to verify. There's nothing you can simply check other than checking the output of 'fips-mode-setup --check' on RedHat to see if it's enabled.
So far I've found for RedHat 8 that you should be using the JDK provided by the OS and not the one that ships with OpenSearch. The one that ships doesn't do anything special on FIPS mode - unless you start configuring it. It doesn't block the non-FIPS implementations which is what starts to cause issues.
Hi @iainjacksonsas, I tried in RedHat8 kernal fips enabled machine with default JDK yes opensearch is not running because default JDK is using SunPKCS11.
But when I use my java not packaged with Opensearch with provider as BouncyCastleFipsProvider its working fine.
Today going one step ahead set up the code locally for opensearch security plugin and changed the provider from BouncyCastleProvider to BouncyCastleFipsProvider in OpenSearchSecurityPlugin.java class and other places. After that also opensearch is starting fine. Can we say now this time its running in FIPS mode ?
Code which I changed. if(Security.getProvider("BC") == null) { Security.addProvider(new BouncyCastleProvider()); // removed this line Security.addProvider(new BouncyCastleFipsProvider()) // added this line }
I initially started working through those steps, but hit into continual roadblocks (cryptography, Java security manager, etc). @szwlhd could you share more details on your environment?
Here's a summary of the environment I have setup and what I have been working with and where I've got to so far:
- RedHat Enterprise Linux 8.3 in FIPS-mode. The VM I'm working with was setup a while ago.
- OpenSearch 2.6 (Haven't updated it yet to 2.7)
- Installed the following libraries into the opensearch/lib directory: bc-fips-1.0.2.3.jar, bcpkix-fips-1.0.7.jar, bctls-fips-1.0.14.1.jar
- JDK from RedHat 8.3, JAVA_HOME=/usr/lib/jvm/jre-11. I also delete the jdk directory when I untar the OpenSearch distribution to ensure I don't use the built-in JDK.
I have the following additions in the security policy and properties files that I used with my no BouncyCastle implementation. I can get OpenSearch to boot with this and it have it running on TLS, but some of the code is still half-assembled and likely doesn't work.
Security Policy
grant {
//io.netty.handler.codec.DecoderException
permission java.lang.RuntimePermission "accessClassInPackage.sun.security.internal.spec";
//java.security.InvalidAlgorithmParameterException: Cannot process GCMParameterSpec
permission java.lang.RuntimePermission "accessDeclaredMembers";
permission org.bouncycastle.crypto.CryptoServicesPermission "tlsAlgorithmsEnabled";
permission org.bouncycastle.crypto.CryptoServicesPermission "exportSecretKey";
permission org.bouncycastle.crypto.CryptoServicesPermission "exportPrivateKey";
permission java.security.SecurityPermission "getProperty.keystore.type.compat";
permission java.security.SecurityPermission "getProperty.jdk.certpath.disabledAlgorithms";
permission java.security.SecurityPermission "getProperty.jdk.tls.disabledAlgorithms";
permission java.security.SecurityPermission "getProperty.jdk.tls.ephemeralDHKeySize";
permission java.security.SecurityPermission "getProperty.jdk.tls.server.defaultDHEParameters";
permission java.security.SecurityPermission "getProperty.jdk.tls.server.enableCAExtension";
permission java.io.FilePermission "${javax.net.ssl.trustStore}", "read";
permission java.io.FilePermission "${java.home}/lib/security/jssecacerts", "read";
permission java.io.FilePermission "${java.home}/lib/security/cacerts", "read";
}
Security Properties
fips.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
fips.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider
fips.provider.3=SUN
fips.provider.4=SunEC
fips.provider.5=SunPKCS11 ${java.home}/conf/security/nss.fips.cfg
fips.provider.6=com.sun.net.ssl.internal.ssl.Provider SunPKCS11-NSS-FIPS
fips.keystore.type=jks
ssl.KeyManagerFactory.algorithm=PKIX
ssl.TrustManagerFactory.algorithm=PKIX
The RedHat JDK has a separate fips.xxx namespace in the security properties for the FIPS-settings which separates them from the non-FIPS settings. These are used instead on a FIPS-enabled system.
Issues Found
The major issues are the following:
- The RedHat JDK is FIPS-enforcing, so code which accesses non-FIPS cryptography via the usual interfaces gets blocked. This is unless you plug the missing cryptography with BC FIPS (or something similar) you run into issues. You run into issues like the keystore can't be created and basic algorithms are missing.
This come baked in with all the FIPS restrictions on what cryptography you can and cannot use. So most of the policy configuration is done. All that remains is to plugin in the BC FIPS provider and get it wired up. If we can get OpenSearch to run in this very strict environment it should prove well for running in others (e.g. Windows, Mac).
- The security plugin also includes BCPROV and you can't have both the FIPS and non-FIPS versions of BouncyCastle on the classpath at the same time even if you have only 1 configured. At the moment it doesn't even get this far with OpenSearch as the JarHell detector picks up on it before it even loads the plugin.
Caused by: java.lang.IllegalStateException: jar hell! class: org.bouncycastle.LICENSE jar1: /home/xxxxxx/opensearch/default/opensearch-2.6.0/plugins/opensearch-security/bcprov-jdk15on-1.67.jar jar2: /home/xxxxxx/opensearch/default/opensearch-2.6.0/lib/bc-fips-1.0.2.3.jar
Trying to have both on the classpath results in weird errors like missing keys, etc when it's initializing as it has the data from both providers from my testing. It's not a valid configuration.
-
The Performance Analyzer plugin also includes BCPROV which conflicts with switching to BC FIPS. This also triggers the JarHell on start-up as well. I removed the plugin from my setup for now.
-
The BCPROV used by the OpenSearch security plugin has hard coded references throughout the security plugin code. Not just to cryptographic or digest functions (e.g. BCrypt, Blake2b) it also uses a bunch of utility classes (Hex, Base64, ASN.1).
Most of these are replaceable using the BC FIPS library instead - except BCrypt and Blake2b (see below).
Switching to BC FIPS also means requiring the BC TLS FIPS and BC PKIX FIPS libraries too. We could ship all of them with the plugin, but ideally you would want to configure this at a JVM level and not through the plugin since these would only get instantiated when the plugin loads.
- The BouncyCastle FIPS library doesn't have BCrypt or Blake2b implementations which are direct references to BouncyCastle in OpenSearch Security.
You can't have both FIPS and non-FIPS dependencies, so if you switch the compilation dependencies to BC FIPS (which is an option) you will be missing a number of classes. I have switched them out locally with other implementations to plug the gap to work through that issue later when investigating.
However, to be running in a FIPS-compatible mode OpenSearch must only use cryptographic algorithms which have been FIPS-certified. So these two algorithms would need to be swapped for an implementation supplied by a FIPS-compliant module e.g. PBKDF2. At the moment there is no method to configure the default password hashing or data anonymization algorithms via settings - this is something that is missing from OpenSearch.
New settings are really needed for:
- Is FIPS Compatible mode enabled? To ensure we don't use non-FIPS algorithms.
- Password Hashing Algorithm. To re-configure the password hashing to use an algorithm from a FIPS-compliant module.
- Data Anonymization Algorithm. Same for the data anonymization algorithm.
Libraries I've been looking at to plug the gaps between BC FIPS and BC:
- https://mvnrepository.com/artifact/at.favre.lib/bcrypt
- https://mvnrepository.com/artifact/com.rfksystems/blake2b
The ASN.1 parsing also needs to be implemented using another library. I've had no luck yet in tracking one down that's usable.
Hi @iaindjackson
Thanks for putting all the details. I completely agree with you we can use BC FIPS and BCPROV together. So I tried by removing BCPROV and including BC-FIPS. After that opensearch started but could not able to run https://localhost:9200 as getting java.lang.NoClassDefFoundError: org/bouncycastle/crypto/generators/OpenBSDBCrypt.
So I am at the place where you mentioned we have to replace the OpenBSDBCrypt algorithms.
I was looking to BC FIPS documentation (https://www.bouncycastle.org/fips-java/BCFipsIn100.pdf) and thinking to implement something below. As of now I am not aware where it can be done or not need to evaluate. But please guide me if investigating in this direction worth it.
Example 6 – ECB Mode Encryption public static byte[] ecbEncrypt(SecretKey key, byte[] data) throws GeneralSecurityException { Cipher cipher = Cipher.getInstance("AES/ECB/PKCS7Padding", "BCFIPS"); cipher.init(Cipher.ENCRYPT_MODE, key); return cipher.doFinal(data); } public static byte[] ecbDecrypt(SecretKey key, byte[] cipherText) throws GeneralSecurityException { Cipher cipher = Cipher.getInstance("AES/ECB/PKCS7Padding", "BCFIPS"); cipher.init(Cipher.DECRYPT_MODE, key); return cipher.doFinal(cipherText); }
Meanwhile I will look into these what you have suggested. https://mvnrepository.com/artifact/at.favre.lib/bcrypt https://mvnrepository.com/artifact/com.rfksystems/blake2b
My environment is
RedHat Enterprise Linux 8.3 in FIPS-mode. OpenSearch 0-SNAPSHOT Installed the following libraries into the opensearch/lib directory: bc-fips-1.0.2.3.jar, bcpkix-fips-1.0.7.jar, bctls-fips-1.0.14.1.jar My own JDK 11, JAVA_HOME=/home/user/java11. I also delete the jdk directory when I untar the OpenSearch distribution..
I my java.security which is at location /home/user/java11/conf/security security.provider.1=org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider C:HYBRID;ENABLE{All}; security.provider.2=org.bouncycastle.jsse.provider.BouncyCastleJsseProvider fips:BCFIPS security.provider.3=SUN security.provider.4=SunRsaSign security.provider.5=SunEC security.provider.6=SunJSSE security.provider.7=SunJCE security.provider.8=SunJGSS security.provider.9=SunSASL security.provider.10=XMLDSig security.provider.11=SunPCSC security.provider.12=JdkLDAP security.provider.13=JdkSASL security.provider.14=SunMSCAPI security.provider.15=SunPKCS11 ssl.KeyManagerFactory.algorithm=PKIX ssl.TrustManagerFactory.algorithm=PKIX keystore.type=pkcs12
Security policy -- there other permission as well but mainly I have added these for BC fips permission org.bouncycastle.crypto.CryptoServicesPermission "tlsAlgorithmsEnabled"; permission java.security.SecurityPermission "putProviderProperty.BCFIPS"; permission java.security.SecurityPermission "insertProvider";
Hi @iaindjackson,
Today I implemented these two libraries https://mvnrepository.com/artifact/at.favre.lib/bcrypt https://mvnrepository.com/artifact/com.rfksystems/blake2b code changed as below -
public boolean passwordMatchesHash(String hash, char[] array) {
return OpenBSDBCrypt.checkPassword(hash, array); // old code
BCrypt.Result result = BCrypt.verifyer().verify(array, hash.getBytes(StandardCharsets.UTF_8)); // newly added
return result.verified;
}
public static String hash(final char[] clearTextPassword) { final byte[] salt = new byte[16]; new SecureRandom().nextBytes(salt); // final String hash = OpenBSDBCrypt.generate((Objects.requireNonNull(clearTextPassword)), salt, 12); final String hash= Bytes.wrap(BCrypt.withDefaults().hash(6,salt,new String(Objects.requireNonNull(clearTextPassword)).getBytes(StandardCharsets.UTF_8))).toString(); Arrays.fill(salt, (byte) 0); Arrays.fill(clearTextPassword, '\0'); return hash; }
private byte[] blake2bHash(byte[] in) { MessageDigest digest =null; try { // final Blake2bDigest hash = new Blake2bDigest(null, 32, null, defaultSalt); // digest = MessageDigest.getInstance("SHA3-224", "BCFIPS"); digest = MessageDigest.getInstance(Blake2b.BLAKE2_B_256); // hash.update(in, 0, in.length); digest.update(in); // final byte[] out = new byte[hash.getDigestSize()]; final byte[] out = new byte[64]; // hash.doFinal(out, 0); // return digest.digest(); // return Hex.encode(out); } catch (NoSuchAlgorithmException e){ System.out.println("Error message "+e); } return digest.digest(); }
code compiled successfully and able to start the opensearch even able to hit https://localhost:9200 and able to login with credential admin/admin
But then getting below error : "javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) at
full error is as below.
Any guidance ?
