bee
bee copied to clipboard
Fetch raw bytes from `bee-tangle`
Description
Introduce a way to fetch raw bytes of certain values using bee-tangle
.
Motivation
Several interfaces inside bee-network
, bee-rest-api
and inx (coming soon!) return the serialized bytes of types like bee_message::Message
. However, both bee-storage
and bee-tangle
only allow us to fetch the deserialized message, meaning that all those interfaces have the following performance pitfall where we deserialize just to serialize again:
Message::unpack_unchecked(bytes).unwrap().pack_to_vec()
Being able to retrieve the raw bytes of a message would avoid this pitfall.
Requirements
- Be able to fetch the bytes of a message from
bee-tangle
.
Open questions (optional)
How do we keep this zero-copy?
We have to guarantee that we aren't copying bytes from the storage unless is strictly necessary.
How do we achieve this?
One option would be to introduce the following type to bee-storage
and implement Fetch<MessageId, Raw<Message>>
for every backend.
pub struct Raw<T> {
bytes: Vec<u8>,
_marker: PhantomData<T>,
}
How does this interact with the cache?
Every iteration of the cache we used inside bee-tangle
stored messages after being deserialized. Introducing this would mean that we need to cache the raw bytes of each message to make it performant. How do we handle this?
- Should we have two caches, one for the deserialized messages and other for the bytes of the messages?
- Should we have a single cache and deserialize on demand?
Can we do this for other types?
It is possible that other types used in the storage could benefit from this.
Are you planning to do it yourself in a pull request?
Yes