tokenizer topic

Naive Bayes text classification implementation as an OmniCat classifier strategy. (#ruby #naivebayes)

mustafaturan

Lindera tokenizer for Tantivy.

lindera

Suika 🍉 is a Japanese morphological analyzer written in pure Ruby

yoshoku

A modern CSS value compiler in JavaScript

robinweser

Fuzzing Parsers with Tokens

uds-se

Converts a string into an array of tokens, with useful methods for looking ahead and behind, capturing, matching, et cetera.

here-be

A PSR-2-R code sniffer and code-style auto-correction-tool - including many useful additions

php-fig-rectified

🛥 Vaporetto is a fast and lightweight pointwise prediction based tokenizer. This is a Python wrapper for Vaporetto.

daac-tools

Thai Natural Language Processing library in Rust, with Python and Node bindings.

PyThaiNLP

Tokenizer (lexer) for golang

bzick