katana icon indicating copy to clipboard operation
katana copied to clipboard

Create navigation and element deduplication mechanism for headless crawler

Open Ice3man543 opened this issue 3 years ago • 0 comments

  • As title says, implement Navigation and Element structures and create their specific deduplication mechanisms

For deduplication, consider element attribute hashing, partial hashing, or similarity hashing. Do benchmarks and choose the best working method.

Ice3man543 avatar Jun 30 '22 08:06 Ice3man543