ScrapySharp
ScrapySharp copied to clipboard
Working Timeout with Proxy
I am scraping a site that requires the use of hundreds / thousands of proxies to make ti through. I have a list of 20,000 proxies, but many are dead. When I encounter a dead proxy, I mark it and retry using a different one, but often times the timeout is around 15 seconds for a dead proxy. It would be nice if setting the browser.Timeout
would handle that on Async calls especially.
Hi @ajhalls I also try to use proxies to access some websites but it seems it does not work for https website. I was wondering if you could show me how to visit https websites using a proxy. Please advise. Thank you in advance.
WebRequest.DefaultWebProxy = new WebProxy("152.26.229.21", 9443);
ScrapingBrowser browser = new ScrapingBrowser();
browser.NavigateToPage(new Uri("https://www.bing.com")); // this does not work