bpe topic

Subword Encoding in Lattice LSTM for Chinese Word Segmentation

jiesutd

Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

rsennrich

Fast and customizable text tokenization library with BPE and SentencePiece support

OpenNMT

Unsupervised text tokenizer focused on computational efficiency

VKCOM

Explains nlp building blocks in a simple manner.

Kyubyong

Byte Pair Encoding for Python!

soaxelbrooke

Machine Learning for Phishing Website Detection

faizann24

Learning BPE embeddings by first learning a segmentation model and then training word2vec

stephantul

JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.

niieani

Ready-made tokenizer library for working with GPT and tiktoken

zurawiki