tokenizer topic

This repository contains the collection of explorative notebooks pure in python and in the language that we, humans can read. Have tried to compile all lectures from the Andrej Karpathy's 💎 playlist...

AayushSameerShah

A character tokenizer for Hugging Face Transformers

dariush-bahrami

Implementation of MambaByte in "MambaByte: Token-free Selective State Space Model" in Pytorch and Zeta

kyegomez

AC自动机 文本相似检索 词库匹配 分词器

aldebran97

A tool that allow you to run SQL-like query on local files instead of database files using the GitQL SDK.

AmrDeveloper

Perfect for anyone who needs to quickly calculate the token amount of ChatGPT in prompts for their project.

liuliuOD

JS tokenizer for LLaMA 3 and LLaMA 3.1

belladoreai

SageMaker Ployglot based RAG opensearch

hyeonsangjeon

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

microsoft

JavaScript Reverse Engineering Toolkit (JSRETK) - Experimental tools for analyzing (minified/obfuscated) JavaScript

SeanPesce