hash calculate in ray deduplicator

[simplew2011]

Before Asking 在提问之前

• [X] I have read the README carefully. 我已经仔细阅读了 README 上的操作指引。

• [X] I have pulled the latest code of main branch to run again and the problem still existed. 我已经拉取了主分支上最新的代码，重新运行之后，问题仍不能解决。

Search before asking 先搜索，再提问

• [X] I have searched the Data-Juicer issues and found no similar questions. 我已经在 issue列表 中搜索但是没有发现类似的问题。

Question

• https://github.com/modelscope/data-juicer/blob/main/data_juicer/ops/deduplicator/ray_document_deduplicator.py#L55
• using md5 to calculate hash in ray
• why not use minhash+lsh and simhash

Additional 额外信息

No response

[simplew2011]

• https://github.com/ChenghaoMou/text-dedup/blob/main/text_dedup/minhash_spark.py

[github-actions[bot]]

This issue is marked as stale because there has been no activity for 21 days. Remove stale label or add new comments or this issue will be closed in 3 day.

[HYLcool]

Hi, thanks for your attention!

We are working on the distributed version of minhash & simhash for Ray mode, which is not very easy though. Please stay tuned.

For now, we already added a dedicated tool for distributed deduplication based on PySpark here. You can try it first.

Feel free to discuss with us if you have any further issues or suggestions!

[github-actions[bot]]

This issue is marked as stale because there has been no activity for 21 days. Remove stale label or add new comments or this issue will be closed in 3 day.

[github-actions[bot]]

Close this stale issue.