playwright-python icon indicating copy to clipboard operation
playwright-python copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

[BUG] Inconsistent header override with Route.continue_

Open elacuesta opened this issue 1 year ago • 0 comments

Context:

  • Playwright Version: 1.28.0
  • Operating System: Linux, macOS
  • Python version: 3.9
  • Browser: Chromium, Firefox, WebKit

Code Snippet

import asyncio
from functools import partial
from playwright.async_api import async_playwright

async def handle_route(route, request, browser_name):
    headers = await request.all_headers()
    print("headers:", headers)
    override_headers = {"cookie": None}
    if browser_name == "firefox":
        override_headers["host"] = "httpbin.org"  # otherwise firefox times out
    await route.continue_(headers=override_headers)

async def run(browser_type):
    print("*" * 100)
    print("browser_type:", browser_type.name)
    browser = await browser_type.launch()
    context = await browser.new_context(
        storage_state={
            "cookies": [{"name": "foo", "value": "bar", "domain": "httpbin.org", "path": "/"}]
        }
    )
    page = await context.new_page()
    await page.route("**", partial(handle_route, browser_name=browser_type.name))
    await page.goto("https://httpbin.org/headers")
    content = await page.locator("pre").text_content()
    print("content:", content)
    await browser.close()

async def main():
    async with async_playwright() as playwright:
        await run(playwright.chromium)
        await run(playwright.webkit)
        await run(playwright.firefox)

if __name__ == "__main__":
    asyncio.run(main())

Describe the bug

Cookies

I can't seem to be able to consistently override cookies in requests using Route.continue_. I'm setting foo=bar as cookie when creating a context, and I'd like to avoid sending it afterwards. What I've tried:

  1. override_headers = {"cookie": None}. I'd expect the cookie header not to be sent, however it is sent by chromium & webkit, not by firefox.
  2. override_headers = {"cookie": "asdf=qwerty"}. I'd expect the cookie header to be "asdf=qwerty", and it is for webkit & firefox, however chromium still sends "foo=bar"
  3. override_headers = {}. Chromium & webkit send "foo=bar", firefox sends no cookie. I have no strong argument for either behavior, the problem for me is the inconsistency.

User agent

Additionally, the user agent header is handled differently by three browsers:

  • Firefox never sends it (I suspect it's because it's never present in the headers dict)
  • Webkit always sends a different value from the one in the request headers
  • Chromium sends the value from the request headers

Sorry for the long snippet ): Please let me know if there's something I'm missing or doing wrong, and thanks in advance for your time.

PS: I'm adding the full outputs below for completeness.

`override_headers = {"cookie": None}` 
****************************************************************************************************
browser_type: chromium
headers: {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'cookie': 'foo=bar', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36'}
override_headers: {'cookie': None}
content: {
  "headers": {
    "Accept-Encoding": "gzip, deflate, br", 
    "Cache-Control": "no-cache", 
    "Cookie": "foo=bar", 
    "Host": "httpbin.org", 
    "Pragma": "no-cache", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "none", 
    "Sec-Fetch-User": "?1", 
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-639d1c70-53698acf1ce28b894044c520"
  }
}

****************************************************************************************************
browser_type: webkit
headers: {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Safari/605.1.15'}
override_headers: {'cookie': None}
content: {
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8", 
    "Cookie": "foo=bar", 
    "Host": "httpbin.org", 
    "User-Agent": "com.apple.WebKit.Networking/615.1.10+ CFNetwork/1335.0.3 Darwin/21.6.0", 
    "X-Amzn-Trace-Id": "Root=1-639d1c71-563620c8772dfef915f7245c"
  }
}

****************************************************************************************************
browser_type: firefox
headers: {'host': 'httpbin.org', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8', 'accept-language': 'en-US,en;q=0.5', 'accept-encoding': 'gzip, deflate, br', 'upgrade-insecure-requests': '1', 'sec-fetch-dest': 'document', 'sec-fetch-mode': 'navigate', 'sec-fetch-site': 'cross-site', 'connection': 'keep-alive', 'cookie': 'foo=bar'}
override_headers: {'cookie': None, 'host': 'httpbin.org'}
content: {
  "headers": {
    "Host": "httpbin.org", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "cross-site", 
    "Upgrade-Insecure-Requests": "1", 
    "X-Amzn-Trace-Id": "Root=1-639d1c74-6e8f740e685efeb55a0ef7d0"
  }
}
`override_headers = {"cookie": "asdf=qwerty"}` 
****************************************************************************************************
browser_type: chromium
headers: {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'cookie': 'foo=bar', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36'}
override_headers: {'cookie': 'asdf=qwerty'}
content: {
  "headers": {
    "Accept-Encoding": "gzip, deflate, br", 
    "Cache-Control": "no-cache", 
    "Cookie": "foo=bar", 
    "Host": "httpbin.org", 
    "Pragma": "no-cache", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "none", 
    "Sec-Fetch-User": "?1", 
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-639d1c8e-466d4e5611e850b919b41e94"
  }
}

****************************************************************************************************
browser_type: webkit
headers: {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Safari/605.1.15'}
override_headers: {'cookie': 'asdf=qwerty'}
content: {
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8", 
    "Cookie": "asdf=qwerty", 
    "Host": "httpbin.org", 
    "User-Agent": "com.apple.WebKit.Networking/615.1.10+ CFNetwork/1335.0.3 Darwin/21.6.0", 
    "X-Amzn-Trace-Id": "Root=1-639d1c90-3405577f7629b0267e6820b7"
  }
}

****************************************************************************************************
browser_type: firefox
headers: {'host': 'httpbin.org', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8', 'accept-language': 'en-US,en;q=0.5', 'accept-encoding': 'gzip, deflate, br', 'upgrade-insecure-requests': '1', 'sec-fetch-dest': 'document', 'sec-fetch-mode': 'navigate', 'sec-fetch-site': 'cross-site', 'connection': 'keep-alive', 'cookie': 'foo=bar'}
override_headers: {'cookie': 'asdf=qwerty', 'host': 'httpbin.org'}
content: {
  "headers": {
    "Cookie": "asdf=qwerty", 
    "Host": "httpbin.org", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "cross-site", 
    "Upgrade-Insecure-Requests": "1", 
    "X-Amzn-Trace-Id": "Root=1-639d1c92-52afd6571cd078e06efd2f53"
  }
}
`override_headers = {}` 
****************************************************************************************************
browser_type: chromium
headers: {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'cookie': 'foo=bar', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36'}
override_headers: {}
content: {
  "headers": {
    "Accept-Encoding": "gzip, deflate, br", 
    "Cache-Control": "no-cache", 
    "Cookie": "foo=bar", 
    "Host": "httpbin.org", 
    "Pragma": "no-cache", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "none", 
    "Sec-Fetch-User": "?1", 
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/108.0.5359.29 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-639d1cbf-3f86b24407e59bbd3c515b00"
  }
}

****************************************************************************************************
browser_type: webkit
headers: {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Safari/605.1.15'}
override_headers: {}
content: {
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8", 
    "Cookie": "foo=bar", 
    "Host": "httpbin.org", 
    "User-Agent": "com.apple.WebKit.Networking/615.1.10+ CFNetwork/1335.0.3 Darwin/21.6.0", 
    "X-Amzn-Trace-Id": "Root=1-639d1cc0-2dd02a8453255d434e1ae214"
  }
}

****************************************************************************************************
browser_type: firefox
headers: {'host': 'httpbin.org', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8', 'accept-language': 'en-US,en;q=0.5', 'accept-encoding': 'gzip, deflate, br', 'upgrade-insecure-requests': '1', 'sec-fetch-dest': 'document', 'sec-fetch-mode': 'navigate', 'sec-fetch-site': 'cross-site', 'connection': 'keep-alive', 'cookie': 'foo=bar'}
override_headers: {'host': 'httpbin.org'}
content: {
  "headers": {
    "Host": "httpbin.org", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "cross-site", 
    "Upgrade-Insecure-Requests": "1", 
    "X-Amzn-Trace-Id": "Root=1-639d1cc2-45d485ac48c417f652180a8c"
  }
}

elacuesta avatar Dec 17 '22 01:12 elacuesta