crossbario
JSON auto-conversion for binary on Python 2
This does work on Python 3 already, but NOT Python 2.
The WAMP spec describes a method to encode binary strings in JSON by prepending the base64 encoding of the binary string with \0. See here.
This isn't implemented. It would need to hook into the string processing with the JSON serializer.
More so, as we target PyPy anyway, we should probably aim for writing our own, bundled JSON thing:
- implements transparent binary bidirectional conversion
- pure Python, optimized for PyPy
We should have more unit tests regarding cross-serializer transparency. I think this is quite important, since it's one of the selling points of WAMP.
I have also found that if I publish binary data with C++ and msgpack, subscribing with autobahn\python\json can stop published events from propagating. If I am subscribed with C++\msgpack I will receive the data, and then sometimes when I subscribe to the topic with autobahn\python\json I will stop receiving events in the c++\msgpack client. In either case I will not receive any events with the autobahn\python\json client. This does not seem deterministic and does not always happen, I am guessing some sort of race.
Ok, implementing this on Python 2.7 requires monkey patching the stdlib, whereas on Python 3 it can be done cleanly: http://stackoverflow.com/questions/31869554/how-can-i-override-behavior-of-jsonencoder-for-binary-data
@urbanpete AutobahnJS has integration tests (against Crossbar.io) for both CBOR and MessagePack:
- https://github.com/crossbario/autobahn-js/blob/master/test/test_serialization_cbor.js
- https://github.com/crossbario/autobahn-js/blob/master/test/test_serialization_msgpack.js
These do work, but you will need to activate CBOR/MessagePack in AutobahnJS explicitly (due to this).
What will not work yet (because of this issue here) is AutobahnJS using JSON, and a different WAMP client using non-JSON with binary app payloads. The auto-conversion isn't implemented, but we'll add that.
For other that run into this, here is the traceback that currently results when Crossbar.io tries to serialize a binary app payload for a WAMP client that is using JSON:
2017-04-04T22:50:42+0200 [Router 27984] WAMP message serialization error: Object of type 'bytes' is not JSON serializable
2017-04-04T22:50:42+0200 [Router 27984] Traceback (most recent call last):
File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/websocket.py", line 119, in send
payload, isBinary = self._serializer.serialize(msg)
File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/serializer.py", line 93, in serialize
return msg.serialize(self._serializer), self._serializer.BINARY
File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/message.py", line 268, in serialize
self._serialized[serializer] = serializer.serialize(self.marshal())
File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/serializer.py", line 187, in serialize
s = _dumps(obj)
File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/serializer.py", line 162, in _dumps
return json.dumps(obj, separators=(',', ':'), ensure_ascii=False)
File "/home/oberstet/cpy361/lib/python3.6/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/home/oberstet/cpy361/lib/python3.6/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/home/oberstet/cpy361/lib/python3.6/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/home/oberstet/cpy361/lib/python3.6/json/encoder.py", line 180, in default
o.__class__.__name__)
TypeError: Object of type 'bytes' is not JSON serializable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/websocket.py", line 95, in onMessage
self._session.onMessage(msg)
File "/home/oberstet/scm/crossbario/crossbar/crossbar/router/session.py", line 536, in onMessage
self._router.process(self, msg)
File "/home/oberstet/scm/crossbario/crossbar/crossbar/router/router.py", line 234, in process
self._dealer.processYield(session, msg)
File "/home/oberstet/scm/crossbario/crossbar/crossbar/router/dealer.py", line 691, in processYield
self._router.send(invocation_request.caller, reply)
File "/home/oberstet/scm/crossbario/crossbar/crossbar/router/router.py", line 197, in send
session._transport.send(msg)
File "/home/oberstet/cpy361_1/lib/python3.6/site-packages/autobahn/wamp/websocket.py", line 123, in send
raise SerializationError(u"WAMP message serialization error: {0}".format(e))
autobahn.wamp.exception.SerializationError: WAMP message serialization error: Object of type 'bytes' is not JSON serializable
One nice recent improvement though is that the client gets its in-flight call properly errored:
(index):57 targs: Array[1]0: "callee disconnected from in-flight request"length: 1__proto__: Array[0]error: "wamp.error.canceled"kwargs: Object__proto__: Object
(index):63 Connection lost: lost
Ok, this is working now on Python 3 (both CPython 3 and PyPy3), merged via https://github.com/crossbario/autobahn-python/pull/808
- Unit tests: https://github.com/crossbario/autobahn-python/blob/master/autobahn/wamp/test/test_serializer.py
- Example: https://github.com/crossbario/crossbar-examples/tree/master/serializers
It does NOT work on CPython 2, hence I leave this issue open as an enhancement.
See the comments in the unit test files and the serializer implementation, as well as above Python issue.
as mentioned above, this works transparently for AB Py3 clients (and hence for CB), but not AB Py2:
With the JSON serializer, this currently only works on Python 3 (both CPython3 and PyPy3),
because even on Python 3, we need to patch the stdlib JSON, and on Python 2, the patching
would be even hackier.
we should at least document that behavior - tbh, it's unlikely we find time for making AB Py2 work on that one.