scrapex icon indicating copy to clipboard operation
scrapex copied to clipboard
verified publisher icon sntran

Avoid re-scrape same URL in one run

Open sntran opened this issue 10 years ago • 0 comments

One run can have duplicate URLs (a product page can belong in multiple categories). Need to avoid re-scraping them.

sntran avatar Jan 03 '16 23:01 sntran