lingua-rs icon indicating copy to clipboard operation
lingua-rs copied to clipboard

Replace json format with more efficient binary file format

Open pemistahl opened this issue 1 year ago • 2 comments

Currently, the language models are stored as json files. Json is somewhat slow to deserialize. Let's investigate whether there is a more efficient binary file format which can be deserialized faster.

A promising candidate could be the MessagePack or Protobuf format.

pemistahl avatar May 29 '23 08:05 pemistahl

What about postcard - Rust?

getreu avatar Jun 10 '23 18:06 getreu

why not use https://github.com/apache/arrow-rs

and

this is Rust serialization benchmark

https://github.com/djkoloski/rust_serialization_benchmark

I recommand https://crates.io/crates/speedy

xxaier avatar Jul 31 '23 06:07 xxaier