django-dynamic-scraper icon indicating copy to clipboard operation
django-dynamic-scraper copied to clipboard

Published 20 hours ago • verified publisher icon holgerd77

Automatically Scrape a HUGE amount of url at scale

Open benjaminelkrieff opened this issue 4 years ago • 0 comments

Hello Everyone.

I am currently working on a project whose goal is to create scrapers in a dynamic way and then being able to process a huge amount of urls at scale.

For example, I have in DDS two websites: www.xxx.it associated with Scraper IT and www.xxx.ca associated with Scraper CA.

I want to send an infinite amount of urls to DDS, and I want all the urls in the form xxx.it/* to be scraped by Scraper IT, and all the urls in the form xxx.ca/* to be scraped by Scraper CA. I want this to be automatically inferred and done by DDS.

Is there a way to achieve this with the current implementation ?

benjaminelkrieff avatar Nov 18 '20 10:11 benjaminelkrieff