test failures OpenSSL.SSL.Error: [('SSL routines', 'SSL_CTX_use_certificate', 'ee key too small')]

[juliantaylor]

In debian unstable with openssl 1.1.1 and CipherString = DEFAULT@SECLEVEL=2 in /etc/ssl/openssl.cnf some tests fail with following error:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 896, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/tmp/buildd/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/negotiate.py", line 316, in debug_fireTimer
    call(*args)
  File "/tmp/buildd/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/negotiate.py", line 526, in sendPlaintextServerAndStartENCRYPTED
    self.startENCRYPTED()
  File "/tmp/buildd/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/negotiate.py", line 555, in startENCRYPTED
    self.startTLS(self.tub.myCertificate)
  File "/tmp/buildd/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/negotiate.py", line 1091, in startTLS
    self.transport.startTLS(ctxFactory)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/_newtls.py", line 179, in startTLS
    startTLS(self, ctx, normal, FileDescriptor)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/_newtls.py", line 139, in startTLS
    tlsFactory = TLSMemoryBIOFactory(contextFactory, client, None)
  File "/usr/lib/python2.7/dist-packages/twisted/protocols/tls.py", line 773, in __init__
    contextFactory = _ContextFactoryToConnectionFactory(contextFactory)
  File "/usr/lib/python2.7/dist-packages/twisted/protocols/tls.py", line 651, in __init__
    oldStyleContextFactory.getContext()
  File "/tmp/buildd/foolscap-0.13.1/debian/python-foolscap/usr/lib/python2.7/dist-packages/foolscap/crypto.py", line 46, in getContext
    ctx = CertificateOptions.getContext(self)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/_sslverify.py", line 1650, in getContext
    self._context = self._makeContext()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/_sslverify.py", line 1660, in _makeContext
    ctx.use_certificate(self.certificate)
  File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 949, in use_certificate
    _raise_current_error()
  File "/usr/lib/python2.7/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'SSL_CTX_use_certificate', 'ee key too small')]

Setting the CipherString SECLEVEL to one fixes the tests. Are there some certificates in the testsuite that might need regenerating with larger keys or newer algorithms?

These tests all fail:

foolscap.test.test_negotiate.Future.testFuture1
foolscap.test.test_negotiate.Future.testFuture2
foolscap.test.test_negotiate.Future.testFuture3
foolscap.test.test_negotiate.Future.testFuture4
foolscap.test.test_negotiate.Parallel.test2
foolscap.test.test_negotiate.Parallel.test3
foolscap.test.test_negotiate.Parallel.test4
foolscap.test.test_negotiate.Parallel.test5
foolscap.test.test_negotiate.Replacement.testAncientClient
foolscap.test.test_negotiate.Replacement.testAncientClientWorkaround
foolscap.test.test_negotiate.Replacement.testBouncedClient
foolscap.test.test_negotiate.Replacement.testBouncedClient_Reverse
foolscap.test.test_negotiate.Replacement.testConnectionHintRace
foolscap.test.test_negotiate.Replacement.testLostDecisionMessage_NewServer
foolscap.test.test_negotiate.Replacement.testNATEntryDropped
foolscap.test.test_negotiate.Replacement.testTwoLostDecisionMessages
foolscap.test.test_negotiate.Replacement.testWeirdSeqnum
foolscap.test.test_negotiate.SharedConnections.test1
foolscap.test.test_negotiate.ThreeInParallel.test1

[warner]

Yeah.. the unit tests use pre-generated certificates (so they'll be repeatable, specifically so that the "who's in charge?" higher-key comparison always gives the same result), and those certificates are deliberately small so that the tests run faster. I think they're probably 512 bits long, far too short for real use but fine for unit tests. The certificate data is in https://github.com/warner/foolscap/blob/foolscap-0.13.1/src/foolscap/test/common.py#L404

I suppose we could regenerate these with larger keys. Computers are faster now than in 2006 when I built them. Any idea what is the shortest/weakest key that modern OpenSSL will accept?

[meejah]

Ah, interesting I'm seeing this same error in Tahoe -- but only under PyPy. My theory is that's because somehow pypy is burning in a newer OpenSSL (or better defaults)

[merkys]

I ran into the same issue again while working to reintroduce foolscap in Debian. In another package I maintain I generate the test keys on the fly this way ensuring they will stay up-to-date. I could in principle do the same for foolscap, but I need to understand a pair of things:

• Why there are two keys? Is it OK if I just generate two distinct keys?
• What is the purpose of tubid_* variables, and how are they related to the contents of certData_*? They look like some sort of checksums, but I may be wrong.

[meejah]

Foolscap makes non-standard use of TLS .. I'll let @warner comment on the details, but I can't immediately think of any reason you couldn't just regenerate the keys. (His note about the "higher" key is I think a comparison of the hashes so just generating two random keys will probably fail about 50% of the time...?)

[merkys]

Thanks for confirming, @meejah. I have attempted regenerating the keys, and most of the tests now pass. Some of the failing tests fail due to:

  File "foolscap/test/test_negotiate.py", line 712, in setUp
    assert self.tub2.tubID > self.tub1.tubID

I guess this is due to different hashes. How can I compare two keys to say which of them is "higher" and which "lower" beforehand?

[meejah]

Not sure of a better way, but one way would be to instantiate a Tub (https://github.com/warner/foolscap/blob/291df027d5770f5edd6df01368de3e2cf273ccdf/src/foolscap/pb.py#L124) with the cert data and ask it for its .tubID .. then compare the two byte-by-byte .. looks like Tub.tubID will be a bytes of length 32. (I think, didn't try).

[merkys]

I have opened PR #90 with new randomly generated keys. With them tests pass.

[meejah]

I think the (only) failure is due to new-enough Python3's refusing to compare None to numbers (whereas earlier versions would do .. something). Other than that, LGTM