text-dedup icon indicating copy to clipboard operation
text-dedup copied to clipboard
verified publisher icon ChenghaoMou

Can we consider using dask for distributed deduplication

Open simplew2011 opened this issue 1 year ago • 0 comments

  • https://github.com/NVIDIA/NeMo-Curator/tree/main/nemo_curator/scripts/fuzzy_deduplication

simplew2011 avatar May 29 '24 09:05 simplew2011