v3.0 KV Protocol Proposal: Strip the first 11 bytes from key in WriteEngine, specify it in ImportEngine

[kennytm]

message ImportEngineRequest {
    bytes uuid = 1;
    string pd_addr = 2;
    bytes key_prefix = 3;
}

Rationale:

An engine currently targets a single table, and contains either only data or only index. This means the first 11 bytes of all keys in an engine are always the same (t«tableID»_r or t«tableID»_i). If a table has 1 billion entries this will waste 11 GB of Lightning → Importer traffic. This also allows the engine be agnostic of table ID, so that the engine file could be reused multiple times.

[kennytm]

Note: this optimization cannot be applied to partitioned tables, where the table ID differ for every partition, unless we store each partition into separate engines (but this will blow up the max-open-engine).

[lance6716]

Is there a uniform routine for normal and partitioned tables? I could only imagine two types of engines for two tables.

[kennytm]

For normal table we set the prefix to be t«tableID»_r, and for partitioned table we set the prefix to be t (or just an empty string).

[overvenus]

Sounds like the prefix compression of RocksDB? https://github.com/facebook/rocksdb/blob/9a449865e3963e7cff0bc7fc7f0731d3dea459de/table/block_based/block_builder.cc#L10-L32

[lance6716]

Sounds like the prefix compression of RocksDB? https://github.com/facebook/rocksdb/blob/9a449865e3963e7cff0bc7fc7f0731d3dea459de/table/block_based/block_builder.cc#L10-L32

I'm not aware of this feature, here we just want to reduce data transfer. Sounds like this feature will reduce concatenate cost and space usage, and maybe have some impact on searching the compressed key? @kennytm

[overvenus]

Actually, this feature is enabled by default.