neural-ir-explorer
neural-ir-explorer copied to clipboard
bling_fire_tokenizer.py: imports fail for Token
Import for the script bling_fire_tokenizer.py fails for the Token object with Python v. 3.9 and 3.8.
Problem lies with the changed allennlp import for the Token object,
Problem is fixed by adding:
from allennlp.data.tokenizers.token_class import Token
from blingfire import text_to_words
s. also image for diff.
BR