byte-pair-encoding topic

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

samber

High performance unsupervised text tokenization for Ruby

ankane

Kotlin multiplatform BPE tokenizer library for OpenAI models

aallam

A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust

gweidart