lingua-rs
lingua-rs copied to clipboard
Replace json format with more efficient binary file format
Currently, the language models are stored as json files. Json is somewhat slow to deserialize. Let's investigate whether there is a more efficient binary file format which can be deserialized faster.
A promising candidate could be the MessagePack or Protobuf format.
What about postcard - Rust?
why not use https://github.com/apache/arrow-rs
and
this is Rust serialization benchmark
https://github.com/djkoloski/rust_serialization_benchmark
I recommand https://crates.io/crates/speedy