vtprotobuf
vtprotobuf copied to clipboard
planetscale
Provide clarification on reusing serialization buffers
Documentation says:
MarshalToVT() ... This function is useful e.g. when using memory pooling to re-use serialization buffers.
Could you please provide clarification on how to use it? I'm asking because protobufs are typically used with gRPC, and grpc-go's SendMsg() returns before the buffer gets put on the wire. Hence, you cannot reuse it. Here is a relevant issue: https://github.com/grpc/grpc-go/issues/2159. Someone has even attempted this http://www.golangdevops.com/2019/12/31/autopool/ but it's not a solution and the finalizers have poor performance. You could find even more details here https://github.com/thanos-io/thanos/pull/4609.
Are there any examples of the usage of this function? I couldn't find anything.
If I am correct then the recommendation in the README seems dangerous.
No, I think you should not use gRPC or the gRPC Go implementation needs to be fixed if you care about pooling on the serialization side. There's some movement here: https://github.com/grpc/grpc-go/issues/6619.
Hey, sorry I've never gotten to this issue before. I think @GiedriusS has already figured this out on his own, but for those reading: the recommendation in the README is unsafe if you're using pooling for the marshal side of protobuf (and I'm going to update it to clarify this). This is a known issue in the way GRPC is implemented in Go, because it keeps ownership of the buffers after they're passed to the RPC call. The tracking issue in https://github.com/grpc/grpc-go/issues/6619 seems to be stalled again, which is slightly disappointing.
I can't think of any good ways to work around this issue until we get upstream changes in GRPC Go.
