tokenizer topic

A library for advanced Natural Language Processing towards multi-modal educational items.

bigdata-ustc

The aim of this repository is to show a baseline model for text classification by implementing a LSTM-based model coded in PyTorch. In order to provide a better understanding of the model, it will be...

FernandoLpz

SQL like 'select' interface for files

SarthakMakhija

A token based HTML Document parser and minifier written in PHP. Extract attribute values and text using CSS selectors.

hexydec

Generic tokenizer written in Nim language 👑 Powered by std/lexbase and Nim's Macros

openpeeps

Lexical analyzer for Javascript developers

zsmoore

A simple tokenizer in Ruby for NLP tasks.

arbox

A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split words, sentences and graphemes.

clipperhouse

A tokenizer based on the dictionary and Bigram language models for Go. (Now only support chinese segmentation)

xujiajun

Promise based streaming tokenizer

Borewit