machinelearning
machinelearning copied to clipboard
[Tokenizers] Port BERTTokenizers
Porting BERTTokenizers enables several text embedding generation models. Requires https://github.com/dotnet/machinelearning/issues/6988.
https://github.com/huggingface/text-embeddings-inference?tab=readme-ov-file#text-embeddings. https://github.com/huggingface/transformers/blob/v4.37.0/src/transformers/models/bert/tokenization_bert.py#L137
cc @luisquintanilla
We already have some BERT implementation which may be sufficient.
Has this been implemented? It seems very similar to #7281.