Cookies retrieved with a custom HOST header don't survive sessions

[Julian]

I'm not 100% sure this is a bug yet, as opposed to a missing feature, but either way:

if you curl something, and use a cookie file, and set a custom host header, e.g.

curl -v -b cookies -c cookies -h HOST my.cool.site.com http://the.example.com

then if the response contains Set-Cookie domain headers curl will be sure to save those and send those along for further requests with the same host.

cookielib however, at least in the way it's currently being used by requests and transitively by httpie, will strip out the headers because they don't match the erhn. (See http://hg.python.org/cpython/file/d1aa8a9eba44/Lib/cookielib.py#l590 which is called by http://hg.python.org/cpython/file/d1aa8a9eba44/Lib/cookielib.py#l977).

And so creating an httpie session does not persist those cookies. E.g. via

http --session test the.example.com HOST:my.cool.site.com

will not store the cookies that are set by the server in the session file.

IIRC I have done similar things when wanting to evade set_ok_domain by mucking with CookieJar subclasses, though cookielib is really bad and hard to work with, but it would be nice if this worked.

[jkbrzt]

I'm not completely sure what the issues is, but the session for this request:

$ http --session=test the.example.com Host:my.cool.site.com

is associated to my.cool.site.com. It's actually equivalent to invoking it like this:

$ http --session=test my.cool.site.com

Note that you can edit/create session files manually (the one created during the above request can be found in ~/.httpie/sessions/my.cool.site.com/test.json).

If you want to use the same session for different hosts, perhaps something like this could do the trick:

ln -s ~/.httpie/sessions/my.cool.site.com/test.json ~/.httpie/sessions/the.example.com/test.json

[Julian]

Hi, thanks for taking a look at this :).

It's actually equivalent to invoking it like this: $ http --session=test my.cool.site.com

I don't think this is true, not for the case I'm asking about. Your code to create a Session doesn't do anything to hook the hostname up to the CookieJar it creates. So any cookies that are set by the server would need to be valid for the hostname in the URL, rather than the one provided in the custom header.

Let me know if it still requires further clarification.

[jkbrzt]

HTTPie uses the session file that you point it to and it doesn't store the domain details on cookies. The hostname (which is only used to find the session file for named sessions) is taken from the URL only if Host hasn't been explicitly set.

$ http --session=cookies-test --print=h 23.23.170.247/cookies/set?hello=world Host:httpbin.org    
HTTP/1.1 302 FOUND
Connection: keep-alive
Content-Length: 223
Content-Type: text/html; charset=utf-8
Date: Mon, 13 May 2013 13:18:54 GMT
Location: http://httpbin.org/cookies
Server: gunicorn/0.16.1
Set-Cookie: hello=world; Path=/    # <== cookie set

$ http --session=cookies-test --print=H 23.23.170.247/get Host:httpbin.org
GET /cookies/get HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, compress
Cookie: hello=world    # <== cookie sent
Host: httpbin.org
User-Agent: HTTPie/0.6.0-dev

$ http --session=cookies-test --print=H  httpbin.org/get
GET /cookies/get HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, compress
Cookie: hello=world     # <== cookie sent
Host: httpbin.org
User-Agent: HTTPie/0.6.0-dev

[Julian]

The example you posted does not set domain.

If you had a

Set-Cookie: hello=world; Path=/; domain=*.example.com

httpie would never even see it to set it (it would get stripped out of the cookie jar before httpie would get a chance to save it into the session file.

[jkbrzt]

Okay, I see now. Not sure if it can be worked around.

[Julian]

The typical workaround is usually to create a more liberal CookieJar (subclass) that doesn't strip out cookies if they match the custom parameters that the request is trying to emulate (i.e. to override the methods I linked in my first post). So cookielib makes it a slight pain, but I thought I'd bring it up since the fact that httpie didn't do it made me go back to cURL to test my app :/.

I'm not sure I'll have some time this week to take a shot at this myself, but if someone else doesn't fix it first at some point I'll try to put a tiny bit of time in next week.

[jkbrzt]

It should actually be quite simple. It's here where the cookie jar gets instantiated: https://github.com/jkbr/httpie/blob/master/httpie/sessions.py#L121

So using a subclass of RequestsCookieJar with the methods overwritten (for both Python 2 and 3) should do the trick.

[Julian]

Slightly complicated by the fact that httpbin sends back a 404 if you send a Host header, so it's non-trivial to test this in the same way as the rest of the code in the area.