third party caveats produce invalid JSON when serialized

[rogpeppe]

trafficstars

It seems that the vid field in third party caveats is not correctly encoded, which leads to an exception when trying to serialize macaroons in JSON format.

>>> m = macaroons.create("my location", "my secret key", "my identifier")
>>> m = m.add_third_party_caveat("tp location", "tp root key", "tp caveat")
>>> m.serialize(format='json')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "bindings/python/macaroons.pyx", line 181, in macaroons.Macaroon.serialize (bindings/python/macaroons.c:2259)
    return bytes(data[:data_sz]).decode('utf8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0x9c in position 90: invalid start byte

I added a print statement before the exception was raised, and it printed this:

('got serialized data', '{"v":2,"l":"my location","i":"my identifier","c":[{"i":"tp caveat","l":"tp location","v":"\x9cJ\xb4jr\x11\xa5\x13\x8e\xba\x13\xe2y\xb4\x829n\x96\x0c1b\x9e\x13\xff\x8a\x89\x10B\xe2\xff+\xf7hh\xe3W\xef\xd3SO\xe0i\x062\xcf\x93\xb1\xc3\xd8r\xa1\xf5\xb5m\\g\xad\xf1\x96\xa4\x164U\xde\t\xb0\x89\x15\xb7\x12"}],"s64":"FodXLpedS2YbvCiUOqNbArA2Edw4Plc7SBFPNHc9gBI"}')

It looks like the vid field should have been base64 encoded and then the v64 field used.

[rescrv]

Yes. As I mentioned in #47, the JSON format wasn't complete w.r.t. UTF-8 and supporting it well. I think at one point I required a build flag stating it was experimental, but may have removed that.

There either is or needs to be a {,de}serialization test for unicode, third party, and strings that must be normalized when converted to unicode.