scrapedin-linkedin-crawler icon indicating copy to clipboard operation
scrapedin-linkedin-crawler copied to clipboard

Published 20 hours ago • verified publisher icon linkedtales

Infinite crawling

Open PriyaJainDev opened this issue 4 years ago • 1 comments

If I need to scroll through, say 10 LinkedIn profile URLs, the crawler after successfully crawling through the 10 URL links provided into the rootProfiles, keep on executing. It keeps on returning the logger as:

2020-05-15T19:53:36.399Z info: starting scraping: undefined show urls undefined 2020-05-15T19:53:36.401Z error: error on crawling profile: undefined TypeError: Cannot read property 'indexOf' of undefined 2020-05-15T19:53:37.400Z info: starting scraping: undefined show urls undefined 2020-05-15T19:53:37.401Z error: error on crawling profile: undefined TypeError: Cannot read property 'indexOf' of undefined 2020-05-15T19:53:38.400Z info: starting scraping: undefined show urls undefined again and again even after profiles has been saved.

Is there any mechanism to stop the script automatically after the number of profile matching and fetching has been done?

Because this infinite crawling might consume up the resource much more than required.

PriyaJainDev avatar May 15 '20 19:05 PriyaJainDev

Yes, you could implement code on crawler.js to check if it has more profiles to crawl

PR is welcomed, thanks

leonardiwagner avatar Jun 03 '20 21:06 leonardiwagner