Exact deduplication

Open Practicinginhell opened this issue 1 year ago • 3 comments

First of all, thank you for providing such an excellent repository. I would like to inquire if the repository supports exact deduplication. Thank you in advance.

Jun 12 '24 13:06 Practicinginhell

Do you mean exact "document" deduplication? As in, remove documents that have their entire content exactly repeated?

Jun 13 '24 08:06 guipenedo

Indeed, that is precisely the point I was intending to convey.

Jun 13 '24 11:06 Practicinginhell

We currently don't support it out of the box. MinHash will also find those documents but that might be overkill if you only want exact matching. Will add to our to do list, but feel free to make a PR if you'd like to work on it

Jun 13 '24 12:06 guipenedo