tokenizer topic

Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.

botisan-ai

Ready-made tokenizer library for working with GPT and tiktoken

zurawiki

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.

dmitry-brazhenko

.NET 7 SDK for OpenAI with a Blazor Server playground

Cledev-Limited

A collection of tools for working with OpenAI

dluc

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

samber

Swift BPE Encoder/Decoder for OpenAI GPT Models. A programmatic interface for tokenizing text for OpenAI ChatGPT API.

alfianlosari

Vietnamese tokenizer (Maximum Matching and CRF)

roy-a

NLP framework: sentence detector, tokeniser, pos-tagger and dependency parser

joliciel-informatique

Smart Language Model

anyks