Urllib3 >1.26.x | Introduces default User-Agent header

[prilly-dev]

trafficstars

With pyrequest pullrequest https://github.com/psf/requests/commit/03957eb1c2b9a1e5e6d61f5e930d7c5ed7cfe853 v2.25.0 urllib3 is updated to v1.26.0. with this version bump urllib3 has changed default behavior in regards to header "User-Agent" : https://pypi.org/project/urllib3/

urllib3 1.26.0

Added default User-Agent header to every request (Pull #1750)
Added urllib3.util.SKIP_HEADER for skipping User-Agent, Accept-Encoding, and Host headers from being automatically emitted with requests (Pull #2018)

Urllib3 now introduces default User-Agent header when no user-agent header is set. Urlib3 also introduces the function "urllib3.util.SKIP_HEADER" as a means to deactivate default user-agent header.

Expected Result

Pyrequest should adopt easy means of deactivating "User-Agent" headers by utilizing the "urllib3.util.SKIP_HEADER" function

Actual Result

Default "User-agent" header is enforced when no header= is specified. by using headers={"User-Agent": none} no longer deactivate default "user-agent" while using headers={"User-Agent": ""} will deactivate "user-agent" header

Reproduction Steps

import requests

req = Request('POST', url=BASE_URL + endpoint, data=params).prepare()  

---> Deafult "user-agent" is enforced "User-Agent: python-urllib3/1.26.2"

req = Request('POST', url=BASE_URL + endpoint, headers={"User-Agent": None}, data=params).prepare()

---> Does not deactivate "User-Agent"

req = Request('POST', url=BASE_URL + endpoint, headers={"User-Agent": ""}, data=params).prepare()

Deactivates "User-Agent" succesfully

## System Information

    $ python -m requests.help

```

This command is only available on Requests v2.16.4 and greater. Otherwise, please provide some basic information about your system (Python version, operating system, &c).

Python 3.8, Docker

[nateprewitt]

Hi @prilly-dev, thanks for bringing this to our attention. I was actually surprised to see Requests wasn't emitting a header here, but it appears this is specific to the Session workflow.

We'll probably need something that is tolerant to urllib3 < 1.26 not being installed. I'm not immediately sure if that means conditionally relying on the urllib3 SKIP_HEADER util in compat or if there are other options. We'll take a look at getting a fix together.

[nateprewitt]

For anyone still trying to work around this issue. The temporary solution is as follows:

import requests
import urllib3

# Where user-agent is the header you want to exclude.
# You can find the full list of "skippable headers" at urllib3.util.SKIPPABLE_HEADERS
headers = {'user-agent': urllib3.util.SKIP_HEADER, ... }

req = Request('GET', someurl)
prep = req.prepare()
prep.headers = headers

s = requests.Session()
resp = s.send(prep)