tokenizer topic

Compress GraphQL Query String

rse

Get identifiers, paths, URLs and words from the previous command output and use them for the next command in xonsh shell.

anki-code

Tooling to play around with multilingual machine translation for Indian Languages.

jerinphilip

A field-tested Hebrew tokenizer for dirty texts (ben-yehuda project, bible, cc100, mc4, opensubs, oscar, twitter) focused on multi-word expression extraction.

eyaler

POS Tagger, lemmatizer and stemmer for french language in javascript

bastienbot

A Python package that do stemming, tokenization, sentence breaking, segmentation, normalization, POS tagging for Arabic language.

disooqi

A simple interpreter for a large subset of Pascal language written for educational purposes

ghaiklor

Lex is an implementation of lex tool in Ruby.

piotrmurach

Morphological analyzer `mystem` (Russian language) wrapper for JVM languages

alexeyev

A very simple python tokenizer for Hebrew text.

YontiLevin