datatrove icon indicating copy to clipboard operation
datatrove copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

MinhashDedup should use the language parameter

Open nrv opened this issue 7 months ago • 0 comments

https://github.com/huggingface/datatrove/blob/1e27cc8819465d5246d89cd929423b76eb0bc5dd/src/datatrove/pipeline/dedup/minhash.py#L196

nrv avatar Jun 29 '24 09:06 nrv