rust-bert
rust-bert copied to clipboard
Add pipelines::masked_language and codebert support
-
Add support for loading local model in SequenceClassificationConfig: Somtimes, the https://huggingface.co/models doesn't contains a rust model (eg. rust_model.ot) of some bert. Hence, it's necessary to load local model easily in SequenceClassificationConfig.
-
Add piplines::masked_language for masking texts more easily; add two examples (masked_language_roberta.rs and masked_language.rs) for utilizing the piplines::masked_language.
-
Add support for codebert; add an example (codebert_example.rs) including on feature_extraction, masked_language and sequence_classification; also upload rust models (rust_model.ot) of huggingface/CodeBERTa-language-id (merged) and microsoft/codebert-base-mlm (waiting to be merged) to https://huggingface.co/models: Rust-bert currently support codebert by a roberta way. Adding codebert support all alone in src/codebert helps inferencing codebert applications more convenient and also is useful for dealing with the open issue (https://github.com/guillaume-be/rust-bert/issues/219).