custom user-agent

[cp2587]

Hello,

We realized recently that altought we had set custom user-agent in headers params of splash/args when doing a request, this header was never used by splash afterward. For now the only working fix we found, is using 'splash:set_user_agent' directly into our lua scripts.

Are you aware of this bug ? Did we miss something ?

[kmike]

Hey @cp2587,

If you're using Lua scripts then headers are not handled automatically. From scrapy-splash readme (meta['splash']['dont_send_headers'] docs):

by default scrapy-splash passes request headers to Splash in 'headers' JSON POST field. For all render.xxx endpoints it means Scrapy header options are respected by default (http://splash.readthedocs.org/en/stable/api.html#arg-headers). In Lua scripts you can use headers argument of splash:go to apply the passed headers: splash:go{url, headers=splash.args.headers}.

Scroll to the last example in the https://github.com/scrapy-plugins/scrapy-splash#examples section - there is an example of how to make headers, cookies, etc. work with a Lua script.

[vionemc]

Does it work with Scrapy fake user agent? https://github.com/alecxe/scrapy-fake-useragent