Make `RedbValue` and `RedbKey` public

[brockelmore]

Currently the type-safe benefits of this library are diminished because a user cannot implement the necessary traits to make a table with their defined types. Instead they currently have to make a table definition of along the lines of TableDefinition<[u8], [u8]> which kinda sucks.

i suggest adding pub use types::{RedbValue, RedbKey}, and whatever else is needed to lib.rs

[cberner]

Ya, I'm planning to add support for custom types! I haven't quite decided on how it should work, so would love your input actually. What's your use case, and how are you thinking of serializing your data?

I could make those traits public and require serialization to/from &[u8]. Alternately, I think I could have users serialize to/from tuples, which avoids them having to implement the serialization logic and may allow for more storage optimizations because the serialization format would be exposed to redb. The new trait would look something like the following (but I think with per trait per desired tuple arity, since it can't be generic over the number of tuple fields):

trait CustomType {
  type SerializedForm: (T0, T1) where T0: RedbValue, T1: RedbValue;

  fn to_serialized(&self) -> Self::SerializedForm;

  fn from_serialized(serialized: Self::SerializedForm) -> Self;
}

struct MyUserStruct {
  name: String,
  age: u8,
}

impl CustomType for MyUserStruct {
  type SerializedForm = (&str, u8);

  fn to_serialized(&self) -> (&str, u8) {
     (&self.name, self.age)
  }

  fn from_serialized(serialized: (&str, u8)) -> Self {
    Self { 
      name: serialized.0.to_string(), 
      age: serialized.1 
    }
  }
}

The other thing I'm waiting for is Generic Associated Types to become stable. Right now the RedbValue trait has to have weird HRTB stuff, like the WithLifetime trait, to work around it.

[brockelmore]

What's your use case, and how are you thinking of serializing your data?

My plan is to use a couple different serialization crates depending on data type, like speedy and serde.

I could make those traits public and require serialization to/from &[u8].

I think this is likely best because it feels like the best UX to just be able to define how to (de)serialize and then have everything work. I relatively strongly dislike the tuple form unless it comes with concrete performance benefits.

[cberner]

Can you give this branch a try? I added the VariableWidthValue trait, which provides a simpler interface than directly exposing RedbValue & RedbKey

[dsobotta]

Thanks for adding that @cberner . I too have a need for user-defined value types. The branch you mentioned looks good, but it's pretty far behind latest now. If you can merge in latest to that branch, I'd be happy to take a look for my project and see how it works out.

Thanks for pulling together this awesome db; performance has been stellar thus far in my tests!

[cberner]

Sure. Try this one: https://github.com/cberner/redb/pull/new/custom_types I think it's more up to date.

Also, can you tell me more about your use-case for user defined types? I'm trying to decide whether they're really necessary, or whether it's reasonable to leave it to users to convert between &[u8] or a tuple type

[dsobotta]

Thanks for the quick response @cberner ! I'll give the linked changes a try later this evening.

It will probably sound bizarre, but I'm utilizing redb as the backend for the compiling and runtime of a custom scripting language. This language defines the gameplay logic and interactions for a multiplayer game.

The general idea is to gather data-driven declarations of variables, transactions, etc and validate/compile them into a cache-friendly and performant form. This is retained in the 'runtime' database as a set of rules or game logic. After compilation, it is read-only and available on both clients and servers. Storing the runtime in this form allows for simple distribution of workload across multiple threads/cores, and minimizes unnecessary coupling of concrete types.

Separate from the 'runtime' db is a 'world' database. It retains information about the layout of the game world and its entities. A small subset of 'world' data is replicated to relevant clients based on the user's interactions with the world (ie. the creature they're fighting, resource node they're harvesting, crafting bench, etc).

Each client also has a corresponding table in a 'user' database, and the entirety of it is replicated to the corresponding client. Reads are done locally to evaluate potential interactions, update UI, etc.. But any actual transactions which affect gameplay are performed on the server. The transaction rules are identical on the client and server's runtime db, but in the event of a desync, the server is already replicating the changes back to re-sync game state.

In terms of actual implementation, I have little need for keys that aren't u128 or str. But I have a long list of user-defined value types that I'd like to store with redb. Here's an example:

pub struct DbsTransaction {
    pub description: String,
    pub conditions: Vec<DbsCondition>, //each entry contains 5-6 vars depending on an internal enum
    pub inputs: Vec<DbsOpSetVar>, //each entry contains 3 vars
    pub outputs: Vec<DbsOpSetVar> //each entry contains 3 vars
}

Manual conversion is obviously possible, but very error prone. Tuples could also quickly get out of hand. Utilizing something like bincode::serialize could alleviate some of the pain; but if the end-user needs to do that for every custom type, it becomes a lot of unnecessary boilerplate.

[dsobotta]

I was using https://github.com/pmagaz/reddb for prototyping early on, and I really appreciate the approach they took of supporting serde-compatible formats for auto-serialization. Obviously, the internals of that crate rely on serde for far more than that; but the interface is nice nonetheless.

If you're concerned about bringing in too many dependencies, perhaps support for something like this can be enabled with a feature flag or addon crate?

[cberner]

Unfortunately, I don't see a way around all the boilerplate, which is why I'm debating whether to actually support custom types or not. It gains you type safety, but I don't think it'll help with boilerplate. It looks like reddb works by copying data during deserialization, which is fine in many cases, but I also want to support zero-copy use-cases.

For your case you might be better off creating a wrapper that uses bincode or another serialization format to convert between your types and a slice.

[dsobotta]

Thanks for the response. Yeah, I wrote a simple proc macro for adding to_bytes() and from_bytes(...) to all relevant structs in the meantime. Type safety is the only major loss at this point.

That said, I do think there is value in exposing the RedbValue and RedbKey types. Doing so locally opened up some options for generics that otherwise would not have been possible.