etcd
etcd copied to clipboard
etcd-io
Strictly define how JSONs are separated in GRPC gateway
I use GRPC gateway to access etcd watching functionality. Namely, https://github.com/etcd-io/etcd/blob/master/Documentation/dev-guide/api_grpc_gateway.md#watch-keys .
But I don't have any specifications on how should I separate JSONs in the response stream since it is not documented anywhere.
Basically I have 3 ways:
-
HTTP chunk = message This fails at least in Fedora 31 on high load because sometimes bare newline
\nappears as a separate chunk. In order to fix you probably should first concatenate JSON with\nbefore calling toWrite(). Also, intermediate proxies may re-chunk stream. -
Messages are separated using strictly one newline. It is not documented as well, and comment in GRPC gateway states that newline is added just to simplify debugging. Somewhere in the net, I found that other software uses a double newline for that purpose (allow to divide messages containing formatted JSON). Currently, I choose this way.
-
Use JSON finite state machine to detect end of a message. This way will work in any case, but technically this requires additional libraries for the majority of programming languages.
So, Please strictly document exact behavior (protocol) which should be considered for implementing JSON stream parser.
Original issue: https://github.com/grpc-ecosystem/grpc-gateway/issues/1086
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.
We encountered the same problem, as there is no clear way to split messages, resulting in the client not being able to handle receiving partial messages or multiple messages in one HTTP chunk when ETCD is proxied by an HTTP server like nginx.
Special attention needs to be paid to ETCD v3.3.0, because of a bug in grpc-gateway v1.3.0(ETCD v3.3.0 depended on), the way of using newlines as separators will fail. See:
- https://github.com/grpc-ecosystem/grpc-gateway/issues/495
- https://github.com/grpc-ecosystem/grpc-gateway/pull/479
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.