dwata icon indicating copy to clipboard operation
dwata copied to clipboard

Integrate with rust-bert

Open brainless opened this issue 9 months ago • 1 comments

As a user I would like to use local libraries to generate embedding. On their website rust-bert is introduced as:

Rust-native state-of-the-art Natural Language Processing models and pipelines. Port of Hugging Face’s Transformers library, using tch-rs or onnxruntime bindings and pre-processing from rust-tokenizers.

We can use this for classification, summarization or other utilities as well in the future.

brainless avatar May 08 '24 16:05 brainless