thanos-io
Investigate alternative encodings for query path
Is your proposal related to a problem?
Thanos currently uses protobuf as the codec for timeseries data on the query path. From profiles I've seen so far, there is a very substantial amount of CPU and memory wasted on marshaling and unmarshaling data, often upward of 50%. This is mostly pronounced in query components which act as store proxies. I have a feeling that protobuf is the wrong format for this type of work and should rather be used for smaller and less frequent payloads.
Describe the solution you'd like
There are several formats that we could start to experiment with which do not have any deserialization cost. They achieve this by having the same in-memory and wire format:
Both of these formats might require writing marshalers by hand, but these messages rarely change and we can gradually move over individual RPCs.
We can also look into Arrow Flight which uses Flatbuffers with some additional tricks to avoid allocations. It is a more opinionated format, but because of that it enables interop with a wider ecosystem.
Describe alternatives you've considered
Getting rid of ZLabels and adding memory pooling through either VTProto or through the gRPC SharedBufferPool is an option, but I have a feeling it solves only half of the problem and won't take as a far as the other approaches.
Additional context
Note that Protobuf != gRPC. All of the above methods still use gRPC with with a different codec for messages in place of protobuf.
We could give this a try with a custom codec that sends flatbuffer over wire and decodes flatbuffers and then populates our proto structs from the flatbuffers so we dont marshal/unmarshal protobuf but flatbuffers yet still can use the protobuf structs in our code? Something like
flatbuffer -> proto request -> Series -> proto response -> flatbuffer
