tokenizer topic

[DISCONTINUED] Source code tokenizer

nette

A WHATWG-compliant HTML5 tokenizer and tag soup parser

untitaker

aim to use JapaneseTokenizer as easy as possible

Kensuke-Mitsuzawa

A tokenizer and sentence splitter for German and English web and social media texts.

tsproisl

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

daac-tools

JavaScript parser written in PHP that generates AST from your code according to ECMAScript specification

mck89

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models

guillaume-be

Collection of developer toolkits

mykolaharmash

Text tokenization and sentence segmentation (segtok v2)

fnl

Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.

mediacloud