cryptography
cryptography copied to clipboard
pyca
37.0.4 regresses wss URIs
Steps to Reproduce
- Use autobahn to connect to a wss:// URI
Expected Result
It connects
Actual Result
2022-08-08T11:15:59-0400 trying transport 0 ("wss://REDACTED") using connect delay 0
2022-08-08T11:15:59-0400 connecting once using transport type "websocket" over endpoint "tcp"
2022-08-08T11:15:59-0400 SSL error: unregistered scheme (in )
2022-08-08T11:15:59-0400 TLS failure: unregistered scheme
Workaround
Downgrade to cryptography==37.0.2
Versions
Versions of Python, cryptography, cffi, pip, and setuptools you're using
cryptography==37.0.4
Python 3.9.13
cffi==1.15.1
pip 22.2.1
setuptools-63.3.0
How you installed cryptography
listed in requirements.txt, pip install -r requirements.txt
autobahn is a very large package, and this report doesn't contain anything that helps us to debug or reproduce this.
37.0.4 contains only bumps in the openssl version versus 37.0.2 (https://cryptography.io/en/latest/changelog/#v37-0-4) so it's very surprising to me that it would break web sockets.
That's fair. I am not an autobahn developer so I don't have much to go on, just like you.
As I was upgrading things individually, upgrading and reverting to the known working autobahn made no difference. What I can say is that, with everything upgraded, the problem occurred and then downgrading only cryptography from 37.0.4 to 37.0.2, leaving everything else on the latest, resolved the problem.
I saw your changelog already and was also confused at how such a small change could regress the behavior autobahn relies on. Do you think it could be on openssl itself?
I can certainly write up a minimal autobahn app to reproduce, but the more difficult aspect would be to have a crossbar server running behind nginx with a cert in place.
In the absence of some way for us to reproduce, it will be very difficult for us to debug and remediate this. A Dockerfile for the server with a self-signed cert should be sufficient.
On Mon, Aug 8, 2022 at 1:50 PM francisATgwn @.***> wrote:
I can certainly write up a minimal autobahn app to reproduce, but the more difficult aspect would be to have a crossbar server running behind nginx with a cert in place.
— Reply to this email directly, view it on GitHub https://github.com/pyca/cryptography/issues/7488#issuecomment-1208426310, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBCFSQTCWEGHGWIUJLDVYFCG7ANCNFSM555W2FFQ . You are receiving this because you commented.Message ID: @.***>
-- All that is necessary for evil to succeed is for good people to do nothing.
Turns out crossbar runs a demo server, so that's not a barrier to reproduction.
However, now I'm confused because my original workaround isn't working. I wonder if something got upgraded on my laptop like openssl.
Currently, this looks like an issue with twisted+pyOpenSSL vs asyncio (which for some reason doesn't require pyOpenSSL).
Works
For definitions of "works" that include an exception due to self-signed server cert
requirements.txt:
autobahn[asyncio]
cbor2
main.py:
from autobahn.asyncio.component import Component
from autobahn.asyncio.component import run
comp = Component(
transports="wss://demo.crossbar.io/ws",
realm="realm1",
)
@comp.on_join
async def joined(session, details):
print("session ready")
if __name__ == "__main__":
run([comp])
Broken
requirements.txt
autobahn[twisted]
cbor2
pyOpenSSL
main.py
from autobahn.twisted.component import Component
from autobahn.twisted.component import run
comp = Component(
transports="wss://demo.crossbar.io/ws",
realm="realm1",
)
@comp.on_join
def joined(session, details):
print("session ready")
if __name__ == "__main__":
run([comp])
As I noted above, unlike my original bug report, adding cryptography==37.0.2 to the requirements.txt does NOT bypass the TLS failure: unregistered scheme error.
Any direction you can provide would be appreciated as I'm a bit out of my depth here. At this point I don't believe this is an issue with your package so feel free to close this issue.
I can replicate the "Unregistered scheme" vs "certificate verify failed" in cryptography 36 vs 37. This corresponds to our transition from OpenSSL 1.1.1 to 3.0.x. Looking at this a bit more the "unregistered scheme" error is logged in autobahn/wamp/component.py and comes from an invocation of handle_connect_error that was originally registered as a txaio callback.
Digging further, OpenSSL 3.0.x has an error called OSSL_STORE_R_UNREGISTERED_SCHEME which returns the "unregistered scheme" text we're seeing and twisted returns the top error on the pyOpenSSL stack. I'm out of time to look at this, but the next step would be understanding how twisted is invoking pyOpenSSL to see if the bug is twisted's (likely) or pyOpenSSL's (also possible, but less likely).
Thanks for digging in and providing some more detail! I'm just bummed that my workaround has changed from "pin cryptography to one rev down" to "migrate off twisted".
Let me know if there's anything else I can do to assist resolving this issue.
You can compile against OpenSSL 1.1.1 yourself as a workaround (a very painful workaround!), but I suspect the twisted folks would like to help with this since it appears to be a serious issue with the latest OpenSSL version. I'd suggest filing an issue with them or joining #twisted-dev on libera.chat to raise the issue and get some traction.
Not sure what's going on, there are a lot of combinations of versions/features of Autobahn/Crossbar possible.
We have a completely self-contained test with
- Crossbar.io with a WebSocket listening endpoint on a TLS socket with a self-signed certificate
- WAMP/WebSocket client using Twisted and ApplicationSession
which you can find here https://github.com/crossbario/crossbar-examples/tree/master/authentication/tls/static
I've tested OpenSSL 1.1.1l with
- cryptography-36.0.2
- cryptography-37.0.4
and both versions do work with above. Here are
- router logs: https://gist.github.com/oberstet/9b57b37059818230b570740ba7942ce4
- client logs: https://gist.github.com/oberstet/670c913419e0c39fd347e0ab88f5e1bc
I've just run another test:
https://github.com/crossbario/crossbar-examples/tree/master/authentication/cryptosign/tls
Also works (with cryptography-37.0.4)
Both tests run TLS on the router side, however, the first test above is authenticating using TLS client certificate authentication, while that second one is using WAMP-Cryptosign (an Ed25519 based method) plus TLS channel binding
@oberstet thanks for consolidating, though I expected the cryptography team will want to close this issue since we have figured out cryptography isn't where the failure is occurring.
Your client cert example works for me also
2022-08-24T07:51:47-0400 initializing component: ComponentConfig(realm=<realm1>, extra={'channel_binding': None, 'authid': None, 'exit_details': None}, keyring=None, controller=None, shared=None, runner=<autobahn.twisted.wamp.ApplicationRunner object at 0x128d9afe0>)
2022-08-24T07:51:47-0400 connected to router
...
2022-08-24T07:51:47-0400 ********************************************************************************
2022-08-24T07:51:47-0400 OK, successfully authenticated with WAMP-cryptosign: realm="realm1", authid="client_0", authrole="backend"
2022-08-24T07:51:47-0400 ********************************************************************************
There's a lot going on in this example so I'm not sure where to go with it. I wanted to try with the minimal requirements.txt in my steps to reproduce, but client_tx.py imports something from crossbar so it's a no go.
I agree with your comment in the ticket I opened in the autobahn-python project, that it's likely upstream. I need some direction to create a minimal reproduction that can be used to open an issue with the relevant upstream. Here's what I am missing:
- is it likely twisted or pyopenssl?
- what code in autobahn can I use as a reference for writing a minimal reproduction using only twisted?
Were you able to reproduce my Actual Result using the Steps to Reproduce I provided? I appreciate the self-contained example, but it's not a minimal reproduction.
And for what it's worth, I'm seeing a much different openssl version than y'all are
>>> OpenSSL.SSL.SSLeay_version(OpenSSL.SSL.SSLEAY_VERSION)
b'OpenSSL 3.0.5 5 Jul 2022'
@oberstet, your second example (TLS cryptosign) also worked for me.
I additionally tried my main.py with pip install crossbar just to see if there is a difference in requirements that may explain the behavior, but I still got the Actual Result (TLS failure: unregistered scheme). So, there is definitely a difference in how the 2 sample applications (your client_tx.py and my main.py) use autobahn+twisted to make the wss connection.
your second example (TLS cryptosign) also worked for me.
very good. as said, this is testing a full client-router TLS, including a custom CA cert chain for the router.
I understand, this is not your setup, but it helps to bisect the problem, as my example allows you to successfully run a working example of what you in principle want to do - on your specific environment (eg openssl version etc)
And for what it's worth, I'm seeing a much different openssl version than y'all are
I just have a stock ubuntu distro with the openssl version that comes with it (and I would not want to change that;)
I need some direction to create a minimal reproduction that can be used to open an issue with the relevant upstream.
here are 2 ideas:
approach 1: take my client_tx.py and add your actual app code into that (stuff from your main.py). does that work probably?
approach 2: use one of our official Autobahn Docker images, and add a test client and server (websocket with TLS only, and own local CA cert hierarchy for server) using autobahn. demonstrate that the problem is indeed triggered
I rewrote the minimal reproduction using the old style ApplicationRunner, as that's what's used by your static TLS authn example.
from autobahn.twisted.wamp import ApplicationRunner, ApplicationSession
class ClientSession(ApplicationSession):
def onJoin(self, details):
print("session ready")
if __name__ == "__main__":
runner = ApplicationRunner(
url="wss://demo.crossbar.io/ws",
realm="realm1",
)
runner.run(ClientSession)
This gives me the Actual Result (SSL error: unregistered scheme), so there's something in the diff between this and your static TLS authn example to explain the behavior.
just couple of more comments .. sorry, I can't allocate resources to track this down:(
- "demo.crossbar.io" is no more (that server is gone). what are you testing?
- the specific certificates involved could trigger the problem
- any intermediaries (like nginx reverse-proxy or what) in your test setup could trigger the problem
"minimally reproducing": can you provide a docker file that reproduces the problem in a self-contained (running on that one container) fashion?
"demo.crossbar.io" is no more (that server is gone). what are you testing?
Oh fun. Something responds there with a self signed cert because the asyncio variant of my minimal reproduction does this:
2022-08-24T08:38:15 trying transport 0 ("wss://demo.crossbar.io/ws") using connect delay 0
2022-08-24T08:38:15 connecting once using transport type "websocket" over endpoint "tcp"
2022-08-24T08:38:16 Connection failed with OS error: SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:997)
I am testing whether or not a wss:// URI is recognized by the framework. In this case, I am considering SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate a success.
the specific certificates involved could trigger the problem
The server-side cert can cause autobahn/twisted/pyopenssl to say that wss is not a recognized URI scheme? It seems to me that the URI would be parsed and its scheme would be determined recognized or not before a TCP socket is even opened.
any intermediaries (like nginx reverse-proxy or what) in your test setup could trigger the problem
Makes sense if there is an issue with the server-side cert that would case the TLS layer to think that wss is not a valid URI scheme. Again, I can't imagine that's the order of operations of the client application.
"minimally reproducing": can you provide a docker file that reproduces the problem in a self-contained (running on that one container) fashion?
I was trying to do this before I discovered the reference to demo.crossbar.io in your documentation. I was having lots of trouble running your crossbar docker image from docker hub with errors about not enough entropy, please install rng-tools. If I could get past that then I would write a Dockerfile that FROMs yours and adds a TLS cert... somehow. Maybe putting nginx in front of crossbar? I dunno. I stopped thinking about once I found demo.crossbar.io.
And to be clear, the minimal reproduction I've been including in these issues is not mine. It's from your autobahn-python documentation.
Oh no! Have I misunderstood the output from the libraries involved?
2022-08-24T08:50:08-0400 trying transport 0 ("wss://demo.crossbar.io/ws") using connect delay 0
2022-08-24T08:50:08-0400 connecting once using transport type "websocket" over endpoint "tcp"
2022-08-24T08:50:08-0400 SSL error: unregistered scheme (in )
2022-08-24T08:50:08-0400 TLS failure: unregistered scheme
I assumed that unregistered scheme refers to wss: in the URI 2 lines above it in the log. Is it a different "scheme" in question?
Dun dun dun! The plot thickens.
crossbar-examples/authentication/tls/static > ./venv/bin/python client_tx.py --key client0.key --cert client0.crt --url wss://demo.crossbar.io/ws
2022-08-24T08:52:51-0400 Connecting to wss://demo.crossbar.io/ws: requesting realm=realm1, authid=None
2022-08-24T08:52:51-0400 TLS client using explicit trust (2 certificates)
2022-08-24T08:52:51-0400 TLS client trust root CA certificate loaded from '/Users/fferrell/code/thirdparty/crossbar-examples/authentication/tls/static/.crossbar/intermediate.cert.pem'
2022-08-24T08:52:51-0400 TLS client trust root CA certificate loaded from '/Users/fferrell/code/thirdparty/crossbar-examples/authentication/tls/static/.crossbar/ca.cert.pem'
2022-08-24T08:52:51-0400 Loaded client TLS key from '/Users/fferrell/code/thirdparty/crossbar-examples/authentication/tls/static/.crossbar/client0.key'
2022-08-24T08:52:51-0400 Loaded client TLS certificate from '/Users/fferrell/code/thirdparty/crossbar-examples/authentication/tls/static/.crossbar/client0.crt' (cn='b'client_0'', sha256=b'40:B7:1C:B6:'..)
2022-08-24T08:52:51-0400 SSL error: certificate verify failed (in )
It seems to me that the URI would be parsed and its scheme would be determined recognized or not before a TCP socket is even opened.
I think the SSL error: unregistered scheme (in ) message actually comes from the openssl library in the client, and the empty string for "in .." might indicate that openssl actually tries to TCP connect to a host that can't be reached at all. Maybe. I don't know. TLS and x509, plus all the Linux and Python and whatnot involved ... lots of fun;)
I was having lots of trouble running your crossbar docker image from docker hub with errors about not enough entropy, please install rng-tools.
docker run -it --rm crossbario/crossbar:cpy-slim-amd64-22.6.1 version
should give you
there are also arm64 and pypy flavors of the image ..
Yes, running the version command gives me that. But running it to start up the router gives:
~ > docker run --rm -it -p 8080:8080 -v ~/.config/crossbar/config.json:/node/.crossbar/config.json crossbario/crossbar:cpy-slim-amd64-22.6.1
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
FATAL: cannot start due to insufficient entropy (11 bytes) available - try installing rng-tools
This is on a Macbook M1. Not sure if that matters.
I'm going to dig in further on the server's cert since now I understand it is actually in play with the unregistered scheme error message.
Wow. This is interesting - for other (unrelated) reasons though;)
I think the FATAL you get is misleading, and the actual thing is the WARNING. Because you are running Macbook M1, which is linux/arm64/v8.
Please try:
docker run -it --rm crossbario/crossbar:cpy-slim-arm64v8-22.7.1.dev1 version
that's an even newer version .. shouldn't make a difference .. we're not baking arm64 automatically for every version yet
also: sorry cryptography team for hijacking and unrelated (crossbar) stuff now .. wasn't my intention:(
The warning appears for every amd64 image I run. It just means it's running through an emulation layer. Many other containers work as expected when that warning apperas.
The image arm64 image you referenced works as expected (the router starts up after removing the version command)
The image arm64 image you referenced works as expected
ok, good.
Many other containers work as expected when that warning apperas.
Running an amd64 Docker image on an arm64 host
- requires the image to bundle qemu static libs
- apparently changes the entropy available in the emulated container
- will be a lot slower
- might run into Qemu emulation issues
and is not necessary, if a native arm64 image is available, as is the case with Crossbar.io
IOW: on arm64, just use our arm64 images (or arm64 snaps) ;)
Starting with the crossbar config in crossbar-examples/authentication/tls/static/.crossbar/config.json, I added an anonymous role (no client-side cert or auth of any kind).
I then started crossbar (./venv/bin/crossbar start) and attempted to connect to it using the minimal example main.py, pointed at wss://localhost:8080/ws. It gives the unregistered scheme.
So, whatever server-side cert ships with crossbar-examples/authentication/tls/static is affected also AND there is something in the diff between my main.py and the example's client_tx.py that works around or solves the problem.
@oberstet docker platform is not the cause of the entropy error
~ > docker run --rm -it -p 8080:8080 crossbario/crossbar:cpy-slim-arm64v8-22.7.1.dev1
FATAL: cannot start due to insufficient entropy (38 bytes) available - try installing rng-tools
Edit to add: consider using docker buildx to build a multi-arch image so the user doesn't have to know their arch or name it in the image tag. Here is an example from my personal work: https://gitlab.com/francisferrell/http-ok
"try installing rng-tools" or "try installing a proper OS"? ;)
crossbar won't start if it cannot make sure (upfront) that it will get enough entropy for security stuff ... it's protecting users from shooting in their feet.
thanks for the docker buildx build hint! we've been using docker manifest annotate in the past for multi-arch images .. but this isn't in the build automation. anyways.
@oberstet I am able to reproduce the error message using your crossbar-examples/authentication/tls/static implementation with the client cert removed. Our production use case does not require TLS client-side certificates for authentication so that piece is just a distraction to me.
diff --git a/authentication/tls/static/.crossbar/config.json b/authentication/tls/static/.crossbar/config.json
index 0710b844..86518b13 100644
--- a/authentication/tls/static/.crossbar/config.json
+++ b/authentication/tls/static/.crossbar/config.json
@@ -97,6 +97,10 @@
"ws": {
"type": "websocket",
"auth": {
+ "anonymous": {
+ "role": "backend",
+ "type": "static"
+ },
"tls": {
"type": "static",
"principals": {
diff --git a/authentication/tls/static/client_tx.py b/authentication/tls/static/client_tx.py
index d5c88c20..d60fc56f 100644
--- a/authentication/tls/static/client_tx.py
+++ b/authentication/tls/static/client_tx.py
@@ -126,7 +126,7 @@ if __name__ == '__main__':
# connect to router and run ClientSession
runner = ApplicationRunner(
- url=options.url, realm=options.realm, extra=extra, ssl=cert_options)
+ url=options.url, realm=options.realm, extra=extra)
runner.run(ClientSession)
# CloseDetails(reason=<wamp.error.not_authorized>, message='WAMP-CRA signature is invalid')
In one terminal: ./venv/bin/crossbar start
In another terminal:
crossbar-examples/authentication/tls/static > ./venv/bin/python client_tx.py --key client0.key --cert client0.crt
2022-08-24T11:39:01-0400 Connecting to wss://localhost:8080/ws: requesting realm=realm1, authid=None
2022-08-24T11:39:01-0400 TLS client using explicit trust (2 certificates)
2022-08-24T11:39:01-0400 TLS client trust root CA certificate loaded from '/Users/fferrell/code/thirdparty/crossbar-examples/authentication/tls/static/.crossbar/intermediate.cert.pem'
2022-08-24T11:39:01-0400 TLS client trust root CA certificate loaded from '/Users/fferrell/code/thirdparty/crossbar-examples/authentication/tls/static/.crossbar/ca.cert.pem'
2022-08-24T11:39:01-0400 Loaded client TLS key from '/Users/fferrell/code/thirdparty/crossbar-examples/authentication/tls/static/.crossbar/client0.key'
2022-08-24T11:39:01-0400 Loaded client TLS certificate from '/Users/fferrell/code/thirdparty/crossbar-examples/authentication/tls/static/.crossbar/client0.crt' (cn='b'client_0'', sha256=b'40:B7:1C:B6:'..)
2022-08-24T11:39:01-0400 SSL error: unregistered scheme (in )
^C2022-08-24T11:39:04-0400 Received SIGINT, shutting down.
2022-08-24T11:39:04-0400 Main loop terminated.
2022-08-24T11:39:04-0400 None
2022-08-24T11:39:04-0400 Traceback (most recent call last):
2022-08-24T11:39:04-0400 File "/Users/fferrell/code/thirdparty/crossbar-examples/authentication/tls/static/client_tx.py", line 135, in <module>
2022-08-24T11:39:04-0400 if extra['exit_details'].reason != 'wamp.close.normal':
2022-08-24T11:39:04-0400 AttributeError: 'NoneType' object has no attribute 'reason'
Note that the CLI options for key and cert are passed just because the argpase default values are invalid.
I also tried being more surgical in removing just the client-side certs, keeping the server-side CA so that the client trusts crossbar's cert:
--- a/authentication/tls/static/client_tx.py
+++ b/authentication/tls/static/client_tx.py
@@ -114,8 +114,6 @@ if __name__ == '__main__':
tls_config = {
"hostname": "localhost",
- "certificate": options.cert,
- "key": options.key,
"ca_certificates": [
"intermediate.cert.pem",
"ca.cert.pem"
This gave a different error:
2022-08-24T11:45:28-0400 Connecting to wss://localhost:8080/ws: requesting realm=realm1, authid=None
2022-08-24T11:45:28-0400 TLS client using explicit trust (2 certificates)
2022-08-24T11:45:28-0400 TLS client trust root CA certificate loaded from '/Users/fferrell/code/thirdparty/crossbar-examples/authentication/tls/static/.crossbar/intermediate.cert.pem'
2022-08-24T11:45:28-0400 TLS client trust root CA certificate loaded from '/Users/fferrell/code/thirdparty/crossbar-examples/authentication/tls/static/.crossbar/ca.cert.pem'
2022-08-24T11:45:28-0400 SSL error: tlsv13 alert certificate required (in )
I'm not sure what is the correct way to configure an autobahn client without client-side certs but with server's trust chain. Regardless, this demonstrates the failure using your own example.
Please advise where I should take this from here.
Hi folks,
Now that we've established that this isn't a pyca/cryptography bug, I'm going to close this out since I think it'd be more appropriate to continue teh debugging elsewhere. (Though there's nothing to stop you from continuing to discuss it on a closed bug!)
Although it may not be a pyca/cryptography bug, I have this error when I try to add a "type": "container" component to my TLS crossbar config, and another SSL error when refreshing the browser (even without the container component).
However, if I leave out the container component the main worker (id anon8080) works without error until I click the refresh button in the browser (chrome on Debian 10) which produces:
SSL error: unexpected eof while reading (in )
Below I posted my transports section and included the component container which reproduces the error in case it helps: , "transports": [ { "id": "anon8080", "type": "web", "endpoint": { "type": "tcp", "port": 443, "tls": { "certificate": "/etc/letsencrypt/live/fqdn/fullchain.pem", "key": "/etc/letsencrypt/live/fqdn/privkey.pem", "dhparam": "dhparam.pem" } }, "paths": { "/": { "type": "static", "directory": ".." }, "ws": { "id": "anon8081", "type": "websocket", "auth": { "wampcra": { "type": "static", "users": { "joe": { "secret": "secret2", "role": "frontend" }, "peter": { "secret": "prq7+YkJ1/KlW1X0YczMHw==", "role": "frontend", "salt": "salt123", "iterations": 100, "keylen": 16 }, "admin": { "secret": "seekrit", "role": "backend" } } } } } } } ], "components": [ { "type": "class", "classname": "hello.AppSession", "realm": "realm1", "role": "frontend" } ] }, { "type": "container", "options": { "pythonpath": [ ".." ] }, "components": [ { "type": "class", "classname": "backend.BackendSession", "realm": "realm1", "transport": { "type": "websocket", "endpoint": { "type": "tcp", "host": "127.0.0.1", "port": 443, "tls": { "hostname": "fqdn", "certificate": "/etc/letsencrypt/live/fqdn/fullchain.pem", "key": "/etc/letsencrypt/live/fqdn/privkey.pem" } }, "url": "wss://fqdn/ws" } } ] } ] }
My crossbar version:
