X509Cert.set_serial breaks for Certificate SerialNumbers with MSB=1

[rhetzler]

in X509Cert.java line 303, set_serial makes a call to org.bouncycastle.x509.X509V3CertificateGenerator.setSerialNumber(BigInteger), which throws an exception if given a negative number.

Sample stacktrace: Java::JavaLang::IllegalArgumentException (serial number must be a positive integer): org.bouncycastle.x509.X509V3CertificateGenerator.setSerialNumber(Unknown Source) org.jruby.ext.openssl.X509Cert.set_serial(X509Cert.java:303) org.jruby.ext.openssl.X509Cert.initialize(X509Cert.java:166) org.jruby.ext.openssl.X509Cert$i$0$1$initialize.call(X509Cert$i$0$1$initialize.gen:65535) ...

( refer to http://bouncycastle.sourcearchive.com/documentation/1.43/classorg_1_1bouncycastle_1_1x509_1_1X509V3CertificateGenerator_23c047ee3c96cb63a0f42cace5333b7b.html )

During initialization, such a certificate with a serial number having MSB=1 can be loaded via java java.security.cert.X509Certificate (line 158), this results in cert.getSerialNumber() (line 166) to yield a negative BigInteger due to it's internal 2's complement representation. Thus, a valid certificate can result in an error.

Typically, such a situation might arise from a certificate authority which generates serial numbers via UID (random bit generation) instead of an incremental approach. This results in a ~50% chance that the MSB=1, and causing this failure. Eg, Serial number "d4 06 2d 6b c9 84 48 86 3d ea c2 cd c1 37 86 8a" (any serial where the first hex digit is 8-f)

The fix is simple, just make sure that the bouncycastle code does not receive a negative SerialNumber. This can be done by forcing BigInteger to interpret the serialNumber bits as positive number by setting the signum in the constructor, as follows:

X509Cert.java:303 -generator.setSerialNumber(bi); +generator.setSerialNumber(new BigInteger(1, bi.toByteArray()));

note: This is difficult to reproduce using certificates generated via openssl CA, since it seems to pad the serial with extra 0s for MSB, neatly avoiding the problem.

Thank you

[sclasen]

Hitting this behavior ATM, was there any decision on what to do here?

[sclasen]

Well that was easy, works for me

[delwaterman]

I am getting this error for jruby 9.0.5.0 (2.2.3) 2016-01-26 7bee00d Java HotSpot(TM) 64-Bit Server VM 25.74-b02 on 1.8.0_74-b02 +jit [darwin-x86_64]

[delwaterman]

Actually ignore my previous comment I misunderstood something with regards to a test.