Merging of default HTTP headers with specified headers breaks the defined ordering

[baderj]

Even if headers are defined in an OrderedDict, their order might change if they coincide with the default_headers set in utils.py, namely User-Agent, Accept-Encoding, Accept, and Connection.

Expected Result

The order of the headers should not change when passed as an OrderedDict. So for example:

import requests
from collections import OrderedDict

headers = OrderedDict([
    ("Accept",   '1'),
    ("Accept-Encoding",   '2'),
    ("User-Agent",   '3'),
])
r = requests.get('http://www.example.com', headers=headers)

should result in the Accept header coming first, followed by Accept-Encoding and finally User-Agent.

Actual Result

The User-Agent comes first, followed by Accept-Encoding and then Accept

Reason

The default headers --- as set by requests.utils.default_headers() --- are merged with the user specified headers by the function requests.sessions.merge_settings. This function first takes the default headers and then updates that with the user specified headers. This ensures that user specified headers overwrite the default ones.

However, the update process does not change the order of the default headers. So when setting User-Agent after Accept, this will not be taken into account as both headers are already in the default headers and in reverse order.

A fix to merge_settings could be to take the user specified headers first, and then iterate over the default headers and add them in if their are misssing.

Workaround

If one wants exactly the headers specified, one can first create a Session and remove the default headers:

s = requests.Session()
s.headers = {}
r = s.get(url, headers=headers, ...)

Having a way to change the default headers on the requests module would be a welcome addition. So one could change, for example, the user agent in one place and have it affect all requests calls that follow.

Reproduction Steps

import requests
from collections import OrderedDict

headers = OrderedDict([
    ("Accept",   '1'),
    ("Accept-Encoding",   '2'),
    ("User-Agent",   '3'),
])
r = requests.get('http://www.example.com', headers=headers)

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "4.0.0"
  },
  "cryptography": {
    "version": "3.3.1"
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.5"
  },
  "platform": {
    "release": "5.8.0-50-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "1010109f",
    "version": "20.0.1"
  },
  "requests": {
    "version": "2.25.1"
  },
  "system_ssl": {
    "version": "1010106f"
  },
  "urllib3": {
    "version": "1.26.4"
  },
  "using_pyopenssl": true
}

[sigmavirus24]

We provide no guarantees of heading ordering because the specification does not require ordering be preserved and we've never attempted to preserve ordering. That turns this into a feature request and the project is under a feature freeze

[baderj]

Unfortunately there are many cases where the ordering of headers are used to detect and block requests from requests. But I understand that requests sticks to the specification and the feature freeze means there should always be the workaround with sessions that can be used.

I'm sorry for the useless issue.

My confusion stems from the usage of OrderedDict, which implies that the header should be sorted. There is even a test case test_headers_preserve_order(self, httpbin) with docstring

Preserve order when headers provided as OrderedDict.

The english documentation explains header orderings very well and that they are not 100% guaranteed: https://docs.python-requests.org/en/master/user/advanced/#header-ordering , but I might have read the German translation, which does not yet have the section on header orderings. I will try to write a pull request for the German version.

Please close the issue

[sigmavirus24]

Hm, I'm surprised we have that test case. I don't know when we changed that but it must be fairly recent. I guess that's a good reason to look into this further if I find the time nad not close it

[biaobro]

useless issue

That's not an useless issue. I found the same issue which makes me confusing too.

There is demand that send request within specified header order.

[biaobro]

I double... triple test my case.... finally found the fixed order is cause my system proxy ... I switched the proxy, then the requested header is following my setting in code

What an embarrassment!