BPCZ
Results
2
comments of
BPCZ
changing self.tokens = tokenized_texts to self.tokens = np.asarray(tokenized_texts) seems to allow token merging I'd submit a PR but I'm not very confident that such a simple thing is the correct...
From what I can tell this project is missing some key scalability features that require big model training to be done on a single GPU with large vram or CPU...