crawlera-tools icon indicating copy to clipboard operation
crawlera-tools copied to clipboard

Published 20 hours ago • verified publisher icon scrapinghub

Specifying extra headers from commind line

Open oltarasenko opened this issue 9 years ago • 25 comments

  • Added a function which generates random ua headers for each request
  • Hardcoded X-Crawlera-UA header for each request

oltarasenko avatar Apr 29 '15 16:04 oltarasenko