HongqiangLi

nltk tokenize doesn't work?

1

Dear Team, The code below doesn't work and the context doesn't sententce token. if args.tokenizer == "PTB": import nltk sent_tokenize = nltk.sent_tokenize def word_tokenize(tokens): return [token.replace("''", '"').replace("``", '"') for token...

When fully functional ?

2

We are excited for the early completion of the development, so that we can have a version that is ready for use.