dwata
dwata copied to clipboard
Integrate with rust-bert
As a user I would like to use local libraries to generate embedding. On their website rust-bert
is introduced as:
Rust-native state-of-the-art Natural Language Processing models and pipelines. Port of Hugging Face’s Transformers library, using tch-rs or onnxruntime bindings and pre-processing from rust-tokenizers.
We can use this for classification, summarization or other utilities as well in the future.