Lais Carvalho

icon indicating a website link laisbsc.github.io icon indicating an email [email protected]

icon company Workday Inc. icon location Dublin, Ireland icon location Software Developer | Director @EuroPython | Member @PyLadiesDublin, @pyjamasconf and co-founder of @MidMeetPy.

Results 1 comments of Lais Carvalho

Obey Robots.txt

> > > ``` > from scrapy.downloadermiddlewares.robotstxt import RobotsTxtMiddleware > from scrapy.http import Request > from twisted.internet.defer import Deferred > > from scrapy.utils.httpobj import urlparse_cached > > > class MyRobotsTxtMiddleware(RobotsTxtMiddleware):...