ipatok icon indicating copy to clipboard operation
ipatok copied to clipboard

additional test data for tokenization tests

Open LinguList opened this issue 7 years ago • 0 comments

I didn't check your data carefull, but this is what we use for lingpy's "ipa2tokens" function:

  • https://github.com/lingpy/lingpy/blob/master/lingpy/tests/test_data/test_tokenization.tsv

You might want to check against those, as lingpy yields 100% tokenizations on them.

LinguList avatar Sep 20 '18 23:09 LinguList