tokenizer topic

C++ Lexer Toolkit Library (LexerTk) https://www.partow.net/programming/lexertk/index.html

ArashPartow

一个微型&算法全面的中文分词引擎 | A micro tokenizer for Chinese

howl-anderson

Parser Building Toolkit for JavaScript

Chevrotain

Solves basic Russian NLP tasks, API for lower level Natasha projects

natasha

Self-contained Japanese Morphological Analyzer written in pure Go

ikawaha

Open Korean Text Processor - An Open-source Korean Text Processor

open-korean-text

Text2Text: Crosslingual NLP/G toolkit

artitw

:herb: NodeJS PHP Parser - extract AST or tokens

glayzzle

CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, t...

CogComp

A Japanese tokenizer based on recurrent neural networks

taishi-i