Question about JSON protocol

[wreed4]

I'm trying to use the TJSONProtocolFactory with the client I'm generating. And from what I can tell, the JSON it is generating is not valid thrift json. For example for a given function call on the client, I'm getting something like this:

'{"metadata": {"version": 1, "name": "registerThing", "ttype": 1, "seqid": 0}, 
"payload": {"request": {"uuid": "12322312121", "url": "someUrl", "user": "JohnSmith", "password": "pass1234"}}}'

In contrast I think I would be expecting something closer to this (the last object part is almost definitely wrong, but that kind of format anyways):

[1, "registerThing, 1, 0, { "1": {"rec" : {"1": {"str": "12322312121"}, "2": {"str": "someUrl"...............

or something like that.

I am very new to Thrift, so I could be wrong here, but I'm confused as to how I'm supposed to use this Protocol. The server I'm trying to connect to is rejecting the request as thriftpy sends it (unfortunately I can't give too much detail on that as it is a proprietary system). I noticed this difference in the json by debugging the request from the client I'm trying to write and comparing it to the requests received from other clients written in other languages.

[ethe]

Could you please post your thrift IDL file here? I will try to reproduce it.

[wreed4]

Unfortunately, I can't paste the thrift file I'm using as it is proprietary. However, here is a very paired-down version of the specific endpoint/type i'm trying to use.

struct Foo {
  1: required string uuid
  2: required string url
  3: required string user
  4: required string password
}

struct FooRequest {
  1: required string uuid
  2: required string url
  3: required string user
  4: required string password
}

struct FooResponse {
  1: required Foo foo
}


service Api {
  FooResponse doFoo(1: FooRequest request)
}

The json it tries to send is

'{"metadata": {"version": 1, "name": "doFoo", "ttype": 1, "seqid": 0}, 
"payload": {"request": {"uuid": "12322312121", "url": "adjdafslkdfkljfad", "user": "23jk", "password": "j23"}}}'

This does not look like the format I expect for thrift.
I'm trying to use TMemoryBuffer (wrote my own factory) and TJSONProtocol.

[ethe]

Well, I tried TJSONProtocol of thriftpy just now, it seems incompatible with Apache Thrift. The request of method you posted above would be serialized to [1,"doFoo",1,0,{"1":{"rec":{"1":{"str":"1"},"2":{"str":"1"},"3":{"str":"sdf"},"4":{"str":"adsf"}}}}](uses TJSONProtocol) or {"request":{"uuid":"1","url":"1","user":"sdf","password":"adsf"}}(uses TSimpleJSONProtocol) with Apache Thrift. I do not know the reason of this incompatibility, But TBinaryProtocol is fully compatible with Apache Thrift. Maybe we can fix TJSONProtocol in the future, thanks!

[wreed4]

Is there a workaround for getting the correct protocol? I tried simply using the TJSONProtocol object from the official thrift module, but to no one's surprise that didn't work.

[ethe]

Use TBinaryProtocol rather than TJSONProtocol, if you want to use both Thriftpy and Apache Thrift at the same time.

[wreed4]

I don't think it's that so much. Rather, the server I'm trying to hit uses TJSONProtocol (it's set up really weird, but basically, it's not actually being sent with thrift, but it's deserializing thrift objects after the fact.. don't ask, it's enterprise software). So short of just rewriting the protocol, I'm not really sure what to do. Is there any way you know of to get around just rewriting the protocol from scratch if the server explicitly requires the TJSONProtocol format?

Sorry if that's a dumb question, I'm not particularly familiar with the thrift landscape.

[ethe]

There are two ways to solve your issue:

1. make your custom protocol
2. use Apache Thrift (TSimpleJSONProtocol or TJSONProtocol)

But, if you just want to deserialze JSON object to Python object, actually you do not need Thrift, there are tons of solutions to realize it:

from __future__ import print_function
import json

try:
    from types import SimpleNamespace as Namespace
except ImportError:
    # Python 2.x fallback
    from argparse import Namespace

data = '{"name": "John Smith", "hometown": {"name": "New York", "id": 123}}'

x = json.loads(data, object_hook=lambda d: Namespace(**d))

print (x.name, x.hometown.name, x.hometown.id)

[wreed4]

That's not my usecase, no. I need to communicate with an xmlrpc server that only accepts TJSONProtocol-style json. So I need to go from a thrift python object, to TJSONProtocol json format, send that to the server, then read TJSONProtocol json back into a thrift python object. The only parts thriftpy doesn't give me apparently is the TJSONProtocol part.

I've tried adapting the Apache Thrift TJSONProtocol object to be able to be used with thriftpy, but that seems like not worth the effort. Currently I'm trying to overwrite enough of thriftpy.TJSONProtocol and surrounding modules that it gives me the correct json. But that's looking more and more like I'd just need to rewrite/adapt most of the protocol since it doesn't seem like TJSONProtocol is all that extensible.

Any ideas you have that are easier are definitely appreciated.

[ethe]

You need to overwrite TJSONProtocol, Apache Thrift TJSONProtocol can not be used with thriftpy.

[JonnoFTW]

I've implemented a JSONProtocol that implements the apache thrift version which also supports binary data (see my fork for the change: https://github.com/Thriftpy/thriftpy2/pull/139

I might not have covered all cases and with python2, but it works in my codebase.