sentencepiece topic

Minimal example of using a traced huggingface transformers model with libtorch

dhpollack

Open source real-time translation app for Android that runs locally

niedev

🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.

himkt

Fast and customizable text tokenization library with BPE and SentencePiece support

OpenNMT

R package for Byte Pair Encoding / Unigram modelling based on Sentencepiece

bnosac

Extremely simple and understandable GPT2 implementation with minor tweaks

Andras7

Learning BPE embeddings by first learning a segmentation model and then training word2vec

stephantul

A Robustly Optimized BERT Pretraining Approach for Vietnamese

nguyenvulebinh

使用sentencepiece中BPE训练中文词表，并在transformers中进行使用。

taishan1994

Rust binding for the sentencepiece library

danieldk