Chris Little

A new PipelineTokenizer() should be added, which takes a list or *args of initialized _Tokenizers. The resulting tokenizers will be called in serial, adding their totals to a common Counter...

Token Levenshtein

1

Is there a variant of Levenshtein that: 1. operates on tokens 2. scores the distance between tokens by another distance measure (possibly Levenshtein itself)

add to notebooks/tutorial

2

add notebooks for the smaller modules (ConfusionMatrix..., Corpus/NGramCorpus, etc.) ~~draw up a brief tutorial for the manual~~ (completed f54e94e)

Try to eliminate: - [ ] encoding errors - [ ] too-long lines - [ ] overfull hboxs (same?) - [ ] underfull hboxs? ....?

implement Cormode's compression distance

1

punted from 0.4.0 -- I can't figure out a good algorithm/recurrence

write a JOSS paper cf. https://joss.theoj.org/

- [ ] Rees/Taxamatch (https://confluence.csiro.au/public/taxamatch/the-rees-2007-phonetic-algorithm-as-used-in-taxamatch) - [x] Taft algorithms (from NYSIIS author's monograph)

http://www.zompist.com/spell.html partial implementation: ` def rosenfelder(word): """Calculate Mark Rosenfelder's English pronunciation of a word. This is based on the rule set/algorithm presented at http://www.zompist.com/spell.html """ # define constants orth_vowels =...

add Kraaij/Pohlmann Dutch stemmer

1

Source: http://snowball.tartarus.org/algorithms/kraaij_pohlmann/stem.c http://snowball.tartarus.org/algorithms/kraaij_pohlmann/stem.h Tests: http://snowball.tartarus.org/algorithms/kraaij_pohlmann/diffs.txt Archive of webpage: http://web.archive.org/web/20010411031043/http://www-uilots.let.uu.nl:80/uplift/

add Henry code

5

French Soundex-type algorithm: https://doi.org/10.3138/CHR-053-04-03 https://www.persee.fr/doc/adh_0066-2062_1976_num_1976_1_1313