surrealdb
Feature: Zero-copy datastore key serialisation and deserialisation
Is your feature request related to a problem?
Currently when creating Vec<u8> keys for use in the datastore, we pass in references, which are then cloned before being serialized. In addition when deserializing a datastore key, the value is cloned to create owned data.
This is unnecessary as the datastore key is never used or held beyond the end of a local function.
Describe the solution
Zero-copy deserialization will ensure that we are not unnecessarily cloning &str values when serializing, and cloning data once again when deserializing.
With this improvement, writing to and reading from the key-value store should be quicker, with less memory allocation.
For this to work, we need to make the storekey deserializer accept borrowed data as can be seen in rmp-serde - https://github.com/3Hren/msgpack-rust/blob/master/rmp-serde/src/decode.rs#L909-L1003
#[derive(Clone, Debug, Eq, PartialEq, PartialOrd, Serialize, Deserialize)]
pub struct Ns<'a> {
kv: &'a str,
_a: &'a str,
ns: &'a str,
}
impl<'a> Into<Vec<u8>> for Ns<'a> {
fn into(self) -> Vec<u8> {
self.encode().unwrap()
}
}
impl<'a> From<Vec<u8>> for Ns<'a> {
fn from(val: Vec<u8>) -> Self {
deserialize(&val).unwrap()
}
}
pub fn new<'a>(ns: &'a str) -> Ns<'a> {
Ns::new(ns)
}
impl<'a> Ns<'a> {
pub fn new(ns: &'a str) -> Ns {
Ns {
kv: BASE,
_a: "!ns",
ns,
}
}
pub fn encode(&self) -> Result<Vec<u8>, Error> {
Ok(serialize(self)?)
}
pub fn decode(v: &[u8]) -> Result<Ns, Error> {
Ok(deserialize(v)?)
}
}
Alternative methods
No alternative methods.
SurrealDB version
surreal 1.0.0-beta.5 for macos on aarch64
Contact Details
No response
Is there an existing issue for this?
- [X] I have searched the existing issues
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
The serializer/deserializer implementation for SurrealDB keys is located in https://github.com/surrealdb/storekey
The keys themselves are all located in https://github.com/surrealdb/surrealdb/tree/main/lib/src/key
Refactoring surrealdb-derive
This cannot be merged yet as it conflicts with Deserialize in storekey crate
Hi there, does this issue still need help? If so, I'd like to work on this.
