tokenization topic

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

nlp-uoregon

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashta...

cbaziotis

Fast and customizable text tokenization library with BPE and SentencePiece support

OpenNMT

Unsupervised text tokenizer focused on computational efficiency

VKCOM

TokenScript schema, specs and paper

SmartTokenLabs

This repository consists of a complete guide on natural language processing (NLP) in Python where we'll learn various techniques for implementing NLP including parsing & text processing and understand...

milaan9

Simple NLP in Rust with Python bindings

rth

All the slides, accompanying code and exercises all stored in this repo. 🎈

AmoDinho

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

adobe

🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer

daac-tools