For Morsel cookies requests expects wrong Expires time format

[druid8]

I'm not sure about this, but it looks that requests is expecting invalid date format in Expires section in Cookies passed as Morsel objects.

morsel_to_cookie (from requests/cookie.py) function parse expires attr if there is no max-age (which is OK) using format from time_template. However this format is set as '%a, %d-%b-%Y %H:%M:%S GMT' which is none of allowed expires section date value described in RFC (https://www.rfc-editor.org/rfc/rfc6265#section-4.1.1).

The nearest format is the first choice from https://www.rfc-editor.org/rfc/rfc2616#section-3.3.1. The difference is that between date parts there should be a space, but requests expects dash.

See Reproduction Steps for very simple example which bases on Python std libs only and crashes.

Expected Result

It's expected that requests properly accepts Morsel cookies when expires follows RFC https://www.rfc-editor.org/rfc/rfc2616#section-3.3.1.

Actual Result

There is ValueError raised by strptime that passed value does not match format. Example: ValueError: time data 'Thu, 01 Jan 1970 00:00:00 GMT' does not match format '%a, %d-%b-%Y %H:%M:%S GMT'

Reproduction Steps

    from http.cookies import SimpleCookie
    from requests.cookies import RequestsCookieJar
    cookies = SimpleCookie()
    cookies.load('auth_session=null; path=/; expires=Thu, 01 Jan 1970 00:00:00 GMT; httponly; samesite=strict')

    jar = RequestsCookieJar()
    jar.update(cookies)

Example above is a simplified case of using async_asgi_testclient to test application written with Starlette with SessionMiddleware. The async_asgi_testclient collects cookies using SimpleCookie class from standard http lib and then pass them to requests lib which is used to perform client test requests.

But this issue is not related to these packages as I reproduced it using only Python standard http lib and requests as above.

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "4.0.0"
  },
  "charset_normalizer": {
    "version": "2.0.9"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.9"
  },
  "platform": {
    "release": "5.15.0-1-amd64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.26.0"
  },
  "system_ssl": {
    "version": "101010cf"
  },
  "urllib3": {
    "version": "1.26.7"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": false
}

[yashd26]

@druid8, I am no expert but after searching for a bit it seems that http.cookies.Morsel follows rfc2109 attributes which expects '%a, %d-%b-%Y %H:%M:%S GMT' as to be its expires attr format

References: https://docs.python.org/3/library/http.cookies.html https://datatracker.ietf.org/doc/html/rfc2109.html#section-10.1.2

[yashd26]

@druid8, I am no expert but after searching for a bit it seems that http.cookies.Morsel (python Morsel module - parent of morsel_to_cookie) follows rfc2109 attributes which expects '%a, %d-%b-%Y %H:%M:%S GMT' as to be its expires attr format

References: https://docs.python.org/3/library/http.cookies.html https://datatracker.ietf.org/doc/html/rfc2109.html#section-10.1.2

Do correct me if am wrong somewhere :)

[druid8]

Correct, however RFC2109 is obsolete and many frameworks and browsers follows RFC2616, none of new async-based web frameworks which I've seen supports old format - all of them generate cookie expire in one of RFC2616 format. Web browsers also expects a 'spaced' format, but old one is and probably for a long time still will be supported: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie#permanent_cookie.

However this issue cause that if someone is using requests library to talk with software which sends cookies in RFC2616 it won't work. Maybe as a fix there should be a set of formats, and library should convert all allowed formats to the one expected by Morsel.

[druid8]

And one more thing, in my reproduction steps the code works as expected until

    jar = RequestsCookieJar()
    jar.update(cookies)

Standard library SimpleCookie accepts passed cookie value and parse it correctly. The issue is later, when RequestsCookieJar() is updated, so maybe conversion is not needed at all.

[yashd26]

And one more thing, in my reproduction steps the code works as expected until

    jar = RequestsCookieJar()
    jar.update(cookies)

Standard library SimpleCookie accepts passed cookie value and parse it correctly. The issue is later, when RequestsCookieJar() is updated, so maybe conversion is not needed at all.

Correct, SimpleCookie.load() already parses the cookie value into morse format whereas when passed to RequestCookieJar all RFC2109 cookies are parsed as RFC2965 or Netscape which I think is causing the issue, this indeed implies conversion is not at all needed.

[mrgrain]

@druid8 Running into the same issue. Did you find a workaround?

[druid8]

@mrgrain I have only a monkey-patch for this. Below is my workaround for pytest, however it can be easily integrated with any other code.

@pytest.fixture(scope='session')
def patch_requests():
    from requests import cookies
    org_mtc = cookies.morsel_to_cookie

    def _patch(value):
        if value['expires']:
            # requests accept invalid datetime format in cookies expires part
            # convert valid RFC formats to expected by requests
            # bug reported: https://github.com/psf/requests/issues/6004
            dt = time.strptime(value['expires'], '%a, %d %b %Y %H:%M:%S GMT')
            value['expires'] = time.strftime('%a, %d-%b-%Y %H:%M:%S GMT', dt)
        return org_mtc(value)

    try:
        cookies.morsel_to_cookie = _patch
        yield
    finally:
        cookies.morsel_to_cookie = org_mtc

[caffeinatedMike]

I just ran into this issue yesterday and was confused as to why this has been a long-standing issue. I think for the sake of backwards compatibility we should try parsing both the new format first, then fallback to the older rfc version upon failure. This will allow older applications to function as expected. Right now I'm doing the following in my code.

import contextlib
from datetime import datetime
from http.cookies import SimpleCookie
import requests

sess = requests.Session()
...
resp = sess.get(...)
# all cookies are stored in a singular Set-Cookie header, so we need to massage them out
# http.cookies.SimpleCookie to the rescue!
cookies = SimpleCookie(resp.headers["set-cookie"])
for item in cookies.items():
    if "expires" in item[1]:
        # account for newer, superseding RFC2616#section-14.21 over RFC2109#section-10.1.2
        with contextlib.suppress(ValueError):
            # if this fails, the application/server is using older RFC2109 expires date standard
            # therefore, we can silently suppress the error.
            item[1]["expires"] = (
                datetime.strptime(item[1]["expires"], "%a, %d %b %Y %H:%M:%S GMT")
            ).strftime("%a, %d-%b-%Y %H:%M:%S GMT")
    sess.cookies.set(*item)

[boltronics]

I know this doesn't help, but it's not even correct for RFC2109 by the looks of it. From section 10.1.2:

Wdy, DD-Mon-YY HH:MM:SS GMT

Requests is using:

        time_template = "%a, %d-%b-%Y %H:%M:%S GMT"

The directives for these are defined here which states:

%y Year without century as a decimal number [00,99]. %Y Year with century as a decimal number.

I wonder if this has ever worked?

$ python3
Python 3.11.5 (main, Sep  2 2023, 14:16:33) [GCC 13.2.1 20230801] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from http.cookies import SimpleCookie
>>> from requests.cookies import RequestsCookieJar
>>> cookies = SimpleCookie()
>>> cookies.load('__cf_bm=truncated_jibberish; path=/; expires=Thu, 28-Sep-23 09:34:34 GMT; domain=.sstatic.net; HttpOnly; Secure; SameSite=None')
>>> jar = RequestsCookieJar()
>>> jar.update(cookies)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.11/site-packages/requests/cookies.py", line 364, in update
    super().update(other)
  File "<frozen _collections_abc>", line 949, in update
  File "/usr/lib/python3.11/site-packages/requests/cookies.py", line 341, in __setitem__
    self.set(name, value)
  File "/usr/lib/python3.11/site-packages/requests/cookies.py", line 219, in set
    c = morsel_to_cookie(value)
        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/requests/cookies.py", line 503, in morsel_to_cookie
    expires = calendar.timegm(time.strptime(morsel["expires"], time_template))
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/_strptime.py", line 562, in _strptime_time
    tt = _strptime(data_string, format)[0]
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/_strptime.py", line 349, in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data 'Thu, 28-Sep-23 09:34:34 GMT' does not match format '%a, %d-%b-%Y %H:%M:%S GMT'
>>>