grpc-ecosystem
Add mapping streaming APIs to newline-delimited JSON streams
This issue copy of https://github.com/cloudendpoints/esp/issues/728
Hi! Please consider adding feature newline-delimited JSON streams like in https://github.com/grpc-ecosystem/grpc-gateway
Example exists in grpc-ecosystem/grpc-gateway#581
This would be useful if api return 100k + lines of statistics on advertising objects without loading everything into memory. =) I really want to use the protocol for exchanging such data, but now, unfortunately, I can’t. Of course, there may be problems with the performance of serialization and deserialization, I am ready to discuss this, maybe it is possible to solve the problem differently.
In general, I would like to have api based on the protocol of buffers, use authorization check at the esp level, also in some api sometimes it is necessary to give a large flow of data (we usually have statistics), while from the external system I already getting stream with 100k + CSV for one request time ( for example Google Ads) and I want to immediately redirect them to the output stream, possibly with minimal processing and json mapping. I don’t want to save data to a cloud guard or something like that and give a link to the data. I want to immediately serialize the answer. Certainly in Google it is already somehow decided, maybe you can tell how to do it correctly from the point of view of Google?
Thank you in advance!
I like to see it is implemented as:
- define a new struct
struct JsonResponseTranslateOption { ::google::protobuf::util::JsonPrintOptions& json_print_options; bool stream_newline_delimted; };
replace this for the last parameter of ResponseToJsonTranslator
-
all implementation should be in this file
-
please add unit_test for it.
You are welcomed to contribute to such change.
@qiwzhang @arturgspb in case anyone's interested, I've taken a stab at implementing this in #73. It's an old thread, but the lack of newline-delimited JSON on streaming endpoints is a major pain point for my own use of gRPC transcoding!
