instascrape
instascrape copied to clipboard
chris-greening
Add proxy options
I am trying InstaScrape but I need to set a proxy (otherwise I have an error). How do I have to proceed?
Hello! What is the error that requires the use of a proxy?
Unfortunately using proxies isn't a feature that is supported out of the box but you can probably brew up your own solution using requests builtin proxy support. instascrape can scrape data from HTML so you can use requests (with your needed proxies) to request data and then pass the response's HTML into an instascrape scraper to scrape the data.
Yes, the point is to prepare "something" that you can add to request . I did it in other projects but the problem within instascrape is that I don't know how to acccess request to set my proxy credentials.
@chris-greening Got some code working to scrape and validate proxies, but am not really sure how your library works. Willing to collaborate on this?
@dtrillo We had to solve this problem as well. What I ended up doing was subclassing the Session object to pass in to Profile.scrape:
class Session(requests.Session):
"""Custom session subclass that allows passing through a timeout to requests.get"""
def __init__(self, timeout=None, proxies=None):
super().__init__()
self.timeout = timeout
self.proxies = proxies
def get(self, url, **kwargs):
kwargs.setdefault('allow_redirects', True)
return self.request('GET', url, **kwargs, timeout=self.timeout, proxies=self.proxies)
and then using it this way:
with Session(timeout=20, proxies=self.proxies) as s:
profile.scrape(session=s)
The above is only necessary for us for authenticated proxies if you want to keep trust_env = True for the Session. Otherwise you can just create a session and add proxies to that without the need for the subclassing
@chris-greening . I am trying to use proxies with InstaScrape to avoid blocking. I tried the solution of @nickhendo but can't get it to work. Any ideas?
