twisted
_sslverify uses md5 for getting key hash
Describe the incorrect behavior you saw
load_entry_point('Twisted', 'console_scripts', 'twistd')()
File "/usr/lib/python3.10/site-packages/twisted/scripts/twistd.py", line 31, in run
app.run(runApp, ServerOptions)
File "/usr/lib/python3.10/site-packages/twisted/application/app.py", line 674, in run
runApp(config)
File "/usr/lib/python3.10/site-packages/twisted/scripts/twistd.py", line 25, in runApp
runner.run()
File "/usr/lib/python3.10/site-packages/twisted/application/app.py", line 381, in run
self.application = self.createOrGetApplication()
File "/usr/lib/python3.10/site-packages/twisted/application/app.py", line 448, in createOrGetApplication
ser = plg.makeService(self.config.subOptions)
File "/usr/lib/python3.10/site-packages/twisted/web/tap.py", line 314, in makeService
svc = strports.service(port, site)
File "/usr/lib/vmware-pod/bin/pod-twistd", line 136, in _monkey_patched_twisted_application_strports_service
endpoints.serverFromString(reactor, description), factory)
File "/usr/lib/python3.10/site-packages/twisted/internet/endpoints.py", line 1731, in serverFromString
nameOrPlugin, args, kw = _parseServer(description, None)
File "/usr/lib/python3.10/site-packages/twisted/internet/endpoints.py", line 1652, in _parseServer
return (endpointType.upper(),) + parser(factory, *args[1:], **kw)
File "/usr/lib/python3.10/site-packages/twisted/internet/endpoints.py", line 1385, in _parseSSL
privateCertificate = ssl.PrivateCertificate.loadPEM(
File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 672, in loadPEM
return Class.load(data, KeyPair.load(data, crypto.FILETYPE_PEM),
File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 650, in load
return Class._load(data, format)._setPrivateKey(privateKey)
File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 633, in _setPrivateKey
if not privateKey.matches(self.getPublicKey()):
File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 768, in matches
return self.keyHash() == otherKey.keyHash()
File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 792, in keyHash
h = md5()
ValueError: [digital envelope routines] initialization error
https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/_sslverify.py#L753
Describe how to cause this behavior
I have enabled openssl fips in my VM and python3 in this VM is built with --with-ssl-default-suites=openssl
And above error message should be enough to reproduce the issue.
What did you do to get it to happen?
I have enabled openssl fips in my VM and python3 in this VM is built with --with-ssl-default-suites=openssl
Does it happen every time you follow these steps, sometimes, or only one time? Every time.
load_entry_point('Twisted', 'console_scripts', 'twistd')()
File "/usr/lib/python3.10/site-packages/twisted/scripts/twistd.py", line 31, in run
app.run(runApp, ServerOptions)
File "/usr/lib/python3.10/site-packages/twisted/application/app.py", line 674, in run
runApp(config)
File "/usr/lib/python3.10/site-packages/twisted/scripts/twistd.py", line 25, in runApp
runner.run()
File "/usr/lib/python3.10/site-packages/twisted/application/app.py", line 381, in run
self.application = self.createOrGetApplication()
File "/usr/lib/python3.10/site-packages/twisted/application/app.py", line 448, in createOrGetApplication
ser = plg.makeService(self.config.subOptions)
File "/usr/lib/python3.10/site-packages/twisted/web/tap.py", line 314, in makeService
svc = strports.service(port, site)
File "/usr/lib/vmware-pod/bin/pod-twistd", line 136, in _monkey_patched_twisted_application_strports_service
endpoints.serverFromString(reactor, description), factory)
File "/usr/lib/python3.10/site-packages/twisted/internet/endpoints.py", line 1731, in serverFromString
nameOrPlugin, args, kw = _parseServer(description, None)
File "/usr/lib/python3.10/site-packages/twisted/internet/endpoints.py", line 1652, in _parseServer
return (endpointType.upper(),) + parser(factory, *args[1:], **kw)
File "/usr/lib/python3.10/site-packages/twisted/internet/endpoints.py", line 1385, in _parseSSL
privateCertificate = ssl.PrivateCertificate.loadPEM(
File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 672, in loadPEM
return Class.load(data, KeyPair.load(data, crypto.FILETYPE_PEM),
File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 650, in load
return Class._load(data, format)._setPrivateKey(privateKey)
File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 633, in _setPrivateKey
if not privateKey.matches(self.getPublicKey()):
File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 768, in matches
return self.keyHash() == otherKey.keyHash()
File "/usr/lib/python3.10/site-packages/twisted/internet/_sslverify.py", line 792, in keyHash
h = md5()
ValueError: [digital envelope routines] initialization error
Preferable a Short, Self Contained, Correct (Compilable), Example on a branch or on a gist.
Automated tests that are demonstrating the failure would be awesome.
Describe the correct behavior you'd like to see A clear and concise description of what you expected to happen, or what you believe should be happening instead.
Testing environment
- Operating System and Version; paste the output of these commands:
root@photon [ ~ ]# uname -a; cat /etc/os-release
Linux photon 5.10.168-4.ph4 #1-photon SMP Thu Mar 2 03:44:05 UTC 2023 x86_64 GNU/Linux
NAME="VMware Photon OS"
VERSION="4.0"
ID=photon
VERSION_ID=4.0
PRETTY_NAME="VMware Photon OS/Linux"
ANSI_COLOR="1;34"
HOME_URL="https://vmware.github.io/photon/"
BUG_REPORT_URL="https://github.com/vmware/photon/issues"
- Twisted version [e.g. 22.2.0] twisted-20.3.0
Additional context None
Also, there are way too many md5 usages in the project. So, not sure how upstream devs want to proceed here.
Hi @sshedi . Many thanks for the report.
Is best to send PRs as small as possible. This will ensure they are easy to review and fast to land.
Twisted development is for now supported by volunteers that are working part-time (after work hours), weekend. We don't have any corporate backing to afford a full time developer dedicated to Twisted.
So rather than having a big PR like "Support FIPS in Twisted", is best to have smaller ones like "Use SHA2 in twisted.internet._sslverity.PublicKey.keyHash()"
Besides the usage of MD5 for SSL/TLS, in Twisted we also have md5 usage in the authentication part smtp, imatp, http digest auth, jabber.
But the most important part here, if we want to support FIPS OpenSSL/LibreSSL is to make sure we have a GitHub Actions job that will run the tests with a FIPS enabled OpenSSL.
@sshedi do you know if there is an publicly available Python 3.10 FIPS and OpenSSL with FIPS container image or some sort that will help us enable the automated tests.
So besides getting the code changes, we also need to update our automated tests and testing infrastructure for FIPS OpenSSL.
@sshedi regarding
I have enabled openssl fips in my VM and python3 in this VM is built with --with-ssl-default-suites=openssl And above error message should be enough to reproduce the issue.
Note that for GitHub Actions we have Ubuntu 22.04 and Ubuntu 20.04 available. Can you provide the steps required to reproduce this error in Ubuntu? With that, we should be able to get the autoamted tests up and running.
It's test to see the code first failing in our automated test suite, then apply the patch and see if fixed :)
I see that the cryptography project are using centos-9-stream with FIPS containers to test this https://github.com/pyca/cryptography/blob/main/.github/workflows/ci.yml#L146
So maybe the python3 from centos-9 is already FIPS enabled and we can use it to run the tests.
Thanks again for the report.
I am happy to work with you in order to get Twisted up and running with a fips OpenSSL
Thanks @adiroiban for your detailed response. I'm happy to work with you to get this done.
AFAIK, VMware Photon OS is the simplest distro to try OpenSSL fips, this distro is fips certified.
To see this feature in actions, do the following:
docker run -it photon /bin/bash
# To enable fips, intall the fips provider package
tdnf install -y openssl-fips-provider
# Verify that fips is effective (the below command should fail)
openssl md5 /etc/os-release
Now go ahead with twisted related tests
Thanks for the info.
I am happy to review and accept a PR for sslverify that removes the usage of md5 and has automated tests executed under a photon container.
Regards
So... a follow-up of the chat discussion
In the example from the PR description you have OpenSSL with FIPS enabled and you try to load a certificate that has a MD5 signature
This is expected to fail, since MD5 is not valid MD5 algorightm
Loading an MD5 cert with FIPS enabled, and somehow automatically make it work is wrong. It will give the users a false confidence.
Also, Twisted is just a framework. It depend to the apps using Twisted and the way the apps are configured to use only FIPS algorightms.
Please add more details for your exact use cases and we can see what can done.
Disabling any MD5 algorightm that is somehow used / enabled by default in Twisted, could be a first step.
Note that Twisted dev team is short on developers.
If your company or end users have the possibility, please consider a donation https://twisted.org/#donate
Regards