tokenizer

tokenizer copied to clipboard

joshweir

The initialization options are ignored. The following splitters could be incorporated: ‘, ’, ”, `

joshweir

Fix #6 Created new splittable `PRE_N_POST_ONLY` which holds characters which can be both prefixes and suffixes but are only a splittable if at the beginning or end of a token...

joshweir

french words that contains single quote get broken down

1

Tokenizer::WhitespaceTokenizer.new.tokenize "et souligne l'interrelation étroite de l'imagerie avec le comportement" => ["et", "souligne", "l", "'", "i", "n", "t", "e", "r", "r", "e", "l", "a", "t", "i", "o", "n", "étroite", "de",...

joshweir

URLs tokenzing individual characters in the URL

1

`Tokenizer::WhitespaceTokenizer.new.tokenize "www.google.com"` ` => ["www", ".", "g","o","o","g","l","e",".","c","o","m"]` I want the website urls to be tokenized as a single noun effectively so would expect www.google.com to tokenize as "www.google.com". I am...

joshweir