ScrapySharp icon indicating copy to clipboard operation
ScrapySharp copied to clipboard

Working Timeout with Proxy

Open ajhalls opened this issue 5 years ago • 1 comments

I am scraping a site that requires the use of hundreds / thousands of proxies to make ti through. I have a list of 20,000 proxies, but many are dead. When I encounter a dead proxy, I mark it and retry using a different one, but often times the timeout is around 15 seconds for a dead proxy. It would be nice if setting the browser.Timeout would handle that on Async calls especially.

ajhalls avatar Sep 24 '19 18:09 ajhalls

Hi @ajhalls I also try to use proxies to access some websites but it seems it does not work for https website. I was wondering if you could show me how to visit https websites using a proxy. Please advise. Thank you in advance.

WebRequest.DefaultWebProxy = new WebProxy("152.26.229.21", 9443);
ScrapingBrowser browser = new ScrapingBrowser();
browser.NavigateToPage(new Uri("https://www.bing.com")); // this does not work

rqg0717 avatar Apr 07 '22 20:04 rqg0717