tokenization topic

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

yooper

Rule-based token, sentence segmentation for Russian language

natasha

🎤 vibrato: Viterbi-based accelerated tokenizer

daac-tools

Minimal, OpenSSL-less and super lightweight JWT library written in C.

GlitchedPolygons

Implementation of the GBST block from the Charformer paper, in Pytorch

lucidrains

Tokenize, encrypt/decrypt, mask your data on the fly with Vaulty proxy

vaulty-co

Multi Cloud Data Tokenization Solution By Using Dataflow and Cloud DLP

GoogleCloudPlatform

Multilingual tokenizer that automatically tags each token with its type

winkjs

An off-the-shelf pre-trained Tweet NLP Toolkit (NER, tokenization, lemmatization, POS tagging, dependency parsing) + Tweebank-NER dataset

mit-ccc

NLP Cheat Sheet, Python, spacy, LexNPL, NLTK, tokenization, stemming, sentence detection, named entity recognition

janlukasschroeder