Eugenio Lacuesta

I still could not reproduce. ```python import scrapy class QuotesSpider(scrapy.Spider): name = "quotes" custom_settings = { "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor", "DOWNLOAD_HANDLERS": { "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler", }, "DOWNLOADER_MIDDLEWARES": { "scrapy.downloadermiddlewares.useragent.UserAgentMiddleware": 400, }, "USER_AGENT": "Mozilla/5.0...

This is a support question, not a bug report of feature request. Please provide a [minimal, reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) (without proxies, additional scripts, processes).

The traceback does not seem to be exactly the same, however the whole situation looks very similar to https://github.com/scrapy/scrapy/issues/5447. I see mentions of `ipython` in your post, I'd recommend trying...

Thanks for the report, I can reproduce with the following spider and a [mitmproxy](https://mitmproxy.org/) instance running locally: ```python from scrapy import Request, Spider class PlaywrightSpiderWithProxy(Spider): name = "proxy-spider" custom_settings =...

This seems to be an upstream thing, I just opened https://github.com/microsoft/playwright/issues/16439 asking about it. There might be a way to work around this by setting cookies in the context before...

Thank you very much for the contribution, but I don't want to include any third-party dependency unless it's really necessary. I've been thinking that one way to allow this functionality...

For the record, this should be possible after #128

> It should be possible to include this with an optional pip dependency e.g. `scrapy-playwright[with_playwright_stealth]` to avoid requiring the dependency while also including this in the distribution That's true, but...

From a quick look, it seems like it might be due to the header processing done by scrapy-playwright. I'd suggest you to look into the `PLAYWRIGHT_PROCESS_REQUEST_HEADERS` setting at https://github.com/scrapy-plugins/scrapy-playwright#supported-settings.

I'm not able to reproduce, the site does not reply to me with a response that matches your code, i.e. no `'consent' in page.url` nor "I agree" button. It could...