neural-ir-explorer icon indicating copy to clipboard operation
neural-ir-explorer copied to clipboard

bling_fire_tokenizer.py: imports fail for Token

Open Genlight opened this issue 3 years ago • 1 comments

Import for the script bling_fire_tokenizer.py fails for the Token object with Python v. 3.9 and 3.8.

Problem lies with the changed allennlp import for the Token object,

Problem is fixed by adding:

from allennlp.data.tokenizers.token_class import Token
from blingfire import text_to_words

s. also image for diff.

BR

grafik

Genlight avatar Jun 18 '21 13:06 Genlight