Convert-PolyAI-Torch
Convert-PolyAI-Torch copied to clipboard
BPE from scratch
implement BPE from scratch with unk tokens hashed (although may achieve worse results on downstream tasks) as # perhaps not as general as bpemb's 25000.model