gaoya icon indicating copy to clipboard operation
gaoya copied to clipboard
verified publisher icon serega

using dask for distributed deduplication.

Open simplew2011 opened this issue 1 year ago • 0 comments

reference：

  • https://github.com/xorbitsai/xorbits/blob/main/python/xorbits/experimental/dedup.py
  • https://github.com/ChenghaoMou/text-dedup/blob/main/text_dedup/minhash_spark.py
  • https://github.com/FlagOpen/FlagData/blob/main/flagdata/deduplication/minhash.py

simplew2011 avatar Jun 06 '24 10:06 simplew2011