klpt icon indicating copy to clipboard operation
klpt copied to clipboard

in_separator in mwe_tokenize doesn't work properly

Open sinaahmadi opened this issue 1 year ago • 0 comments

The argument in_separator seems to be overwritten at the end of the mwe_tokenize function. The argument is not accessible through word_tokenizer either.

sinaahmadi avatar Jun 18 '23 01:06 sinaahmadi