katana
katana copied to clipboard
Create navigation and element deduplication mechanism for headless crawler
- As title says, implement Navigation and Element structures and create their specific deduplication mechanisms
For deduplication, consider element attribute hashing, partial hashing, or similarity hashing. Do benchmarks and choose the best working method.