Scrapy's USER_AGENT != Selenium user-agent

[ohld]

hey 👋

I just realized that Scrapy's USER_AGENT from settings.py doesn't equal Selenium's user_agent. I expect that scrapy-selenium will take that value and pass to driver's options.

I solved the issue with these lines in settings.py:

SELENIUM_DRIVER_ARGUMENTS = [
    '--headless',
    f'--user-agent={USER_AGENT}'
]

This is ok for now, but what if I would like to use scrapy_fake_useragent plugin to generate different User agents each time? In that case, my trick will not work.

I suppose, `scrapy-selenium' should add scrapy's USER_AGENT to driver's options automatically (maybe somewhere in middleware).