tokenizer topic

Persian NLP Toolkit

roshan-research

A collection of NLP tools for Sinhalese (සිංහල).

ysenarath

Nand2Tetris: Build a computer system from the ground up, from nand to tetris. Hardware and software.

VanTamNguyen

Cereja is a bundle of useful functions we don't want to rewrite and .. just pure fun!

cereja-project

Online playground for OpenAPI tokenizers

dqbd

⚙️ A simple tokenizer for deno

denosaurs

使用 PaddlePaddle 实现基于深度神经网络的中文分词引擎 | A DNN Chinese Tokenizer by Using PaddlePaddle

howl-anderson

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

alasdairforsythe

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (gpt-5, gpt-o*, gpt-4o, etc.). Port of OpenAI's tiktoken with additional features.

niieani

JS tokenizer for LLaMA 1 and 2

belladoreai