[OEP16] Binary protocol review

[tglman]

Summary: A complete review of the binary protocol, introducing streaming, server side transaction, and specifications for any other data send on the binary channel.

Goals: Stabilize the binary protocol introducing all features needed for all the 3.x version and future, avoiding the need of review.

Non-Goals:

Define a replacement of the binary protocol.

Success metrics:

No change of the specification for all the 3.x version. Support of this specifications from third party.

Motivation:

Stabilization of the binary protocol. Add support for the functionality requirements for 3.0

Description:

For version 3.0 will be a general refactor on the binary protocol, it will guarantee the compatibility with 2.x protocol.

The changes are split in 6 main area

1. general review of protocol flow, handshaking and low basic data transfer
2. Redesign of push request based on subscribe.
3. Introduction of server side transaction
4. Introduction of new result set handling
5. Unification under the same protocol specification of all client required informations
6. Review of error format and specification of error by code

1) Genera review of protocol

• Introduction of a new phase of handshaking as soon as a connection happen where client and server share protocol versions and specifications.
• remove all the protocol specific information from connect and open
• evaluate if introduce varint in the serialization of network messages.

2) Subscribe based Push Request

The push request flow will change to the following steps:

• redefine the push request flow with event subscribe, and client reception ack.
• redefine the push messages for distributed configuration, schema structure, database structure(like list of clusters), live query

3) Server Side transaction

• Introduction of a new begin transaction, and "rebegin transaction" with updated data
• add support of transaction download and upload after a not idempotent query
• Introduce of commit of a current running transaction

4) Introduction of new result set handling

• Define a new pagined query result set, in a stateful way.
• Define serialization of query result with support of projection, nested projection and record with vertex and edge specialization.
• Define message for close the query, interrupt it before end.

5) Unification under the same protocol specification of all client required informations

• Integrate the record serialization format in the protocol, making it to follow the protocol versions.
• Define specification of storage metadata.
• Define specification of distributed configuration information for clients.
• Define specification of database metadata (Schema, Index, Functions, Security,Scheduler, Sequences) information for clients.

6) Review of error format and specification of error by code

• Introduce of different error serialization format as a replacement of current java exception serialization
• Specify error codes for all the major kind of errors, and user interaction errors.

Alternatives:

Define a new protocol, that cover all the case that we cover now with all the new cases.

Risks and assumptions:

Is going to require an big effort from contributors.

Impact matrix

• [ ] Storage engine
• [ ] SQL
• [x] Protocols
• [ ] Indexes
• [ ] Console
• [ ] Java API
• [ ] Geospatial
• [ ] Lucene
• [x] Security
• [ ] Hooks
• [ ] EE

[luigidellaquila]

Just a couple of comments:

1) Genera review of protocol

consider that this protocol is also used by the distributed layer for inter-server communication (maybe it's a good time split the two?)

3) Server Side transaction

add support of transaction download and upload after a not idempotent query (When a statement is executed on the server, the client transaction has to be flushed to the server, but also the server transaction has to be sent back to the client)

[tglman]

1. yes i want to distributed be out of the spec, and probably that speck don't need to be public.
2. yes i meant that, corrected

[tglman]

Hi,

Most of this has been implemented, working on the documentation and defining the push for metadata. we are near to close this OEP with the full implementation