awesome-align icon indicating copy to clipboard operation
awesome-align copied to clipboard

Published 20 hours ago • verified publisher icon neulab

added multiprocessing to LineByLineTextDataset class

Open vigneshmj1997 opened this issue 3 years ago • 0 comments

Added multiprocessing to LineByLineTextDataset class since tokenizer.prepare_for_model takes lot to time to process for large datasets

vigneshmj1997 avatar Jan 07 '22 14:01 vigneshmj1997