GetRecords response with all logs in the thread

[dgtony]

Core API method GetRecords is currently used in two cases:

• Pulling the thread. Here we expect receiving information about all the logs including ones we are not aware of yet.
• Node receives PushLog request, containing updated information about some specific log. After that recipient starts pulling specified log from the sender (btw why always pull log from the beginning?), but currently it will receive entire thread information including records for all the logs of the thread in response.

Looks like there is some redundancy, and network traffic and resource consumption can be optimized here. For example, we can add special flag allLogs in protobuf message GetRecordsRequest and implement it on the server. Method server.GetRecords by default should return only log IDs explicitly specified in the request and setting the flag allLogs = true enables current behaviour, when server returns all known logs associated with the given thread ID.

@sanderpick what do you think about such a modification?

[sanderpick]

The current setup is pretty heavy-handed. I think what you propose here sounds like a great idea.

btw why always pull log from the beginning?

We should only be pulling new logs from the beginning. Existing logs are pulled from an offset specified by the client.

[dgtony]

Unfortunately, it turns out we cannot implement it due to some limitations of protocol design. Despite the backward compatibility support in protobuf, current scheme doesn't allow different versions to communicate. Let's say there are two versions of communication protocol v1 and v2 with minor difference (like a single additional field) and two nodes: A (v1) and B (v2) trying to communicate. Every node on receiving a request verifies it at first. It takes provided signature and public key from the header and checks if body has the same signature with given key. But server/client methods operate on a deserialized protocol structures, as gRPC makes all the encoding job for us. Hence in order to ensure payload consistency both sender and receiver serialize it into a bytestring. And here is the problem: node A receiving message from B will be able to fit it into the previous format thanks to protobuf, but serializing it back into bytestring will give result different from the original payload. Therefore signature doesn't match and request will be rejected. Looks like an obstacle for the seamless evolution of protocol, isn't it?

[sanderpick]

Hrm. Yeah tricky. In fact, I have been wanting to remove those signatures altogether as they are no longer needed: https://github.com/textileio/go-threads/issues/328

We could do that, then bumped the protocol version and move forward with a more upgradable protocol.