Search-Engines-Scraper icon indicating copy to clipboard operation
Search-Engines-Scraper copied to clipboard

Published 20 hours ago • verified publisher icon tasos-py

page endpoint

Open sushidelivery opened this issue 4 years ago • 2 comments

Hello! Thank you for your amazing contribution!

I would wanted to ask about using proxy with email & password? How can I configure that?

Many thanks!

i got it. Just enter like this "http://user:[email protected]:3128/" Please, reply on the second question;)

sushidelivery avatar Jun 17 '21 05:06 sushidelivery

and second question, is there anyway I can add a checkpoint, so if i scraped for first time, and then when i scrape for the second time it will start from page where first time ended?

sushidelivery avatar Jun 17 '21 06:06 sushidelivery

No, there is no starting page option. You could implement this feature, and I would help, but it won't be easy. You would have to modify the ._next_page() method of every class that inherits from SearchEngine separately, because you would have to be able to ask for a specific pagination element and each search engine has different HTML structure.

tasos-py avatar Jun 20 '21 22:06 tasos-py