Add form and multipart_form helpers to request

[jamielennox]

We have a json() helper on last_request, however for form encoded a=b&c=d data and multipart form encoded data we don't have a helper. Whilst there are less useful with APIs they are a common usage for people using requests.

Launchpad Details: #LP1632584 Jamie Lennox - 2016-10-12 07:28:51 +0000

[jamielennox]

Might I ask why? Is it to ensure that requests encoded it properly? That seems more like someone testing requests than their code imo

Launchpad Details: #LPC Ian Cordasco - 2016-10-12 12:02:17 +0000

[jamielennox]

Sure, i consider that a major advantage of the library is not just stubbing out responses but being able to analyze what you sent. This is something I know i use a lot.

For example we have a big request() wrapper function that adds certain things, after you call it you can do:

last_request = self.requests_mock.last_request

self.assertEqual(val, last_request.headers[key]) self.assertEqual(dict_data, last_request.json())

the JSON one gets used a lot, and it's not that we don't trust that requests has done the right thing it's mostly that by passing some flag or param to a function i invoke the change i expect to receive in the outputted message.

Because I'm mainly using json apis I haven't really used the:

requests.post(url, data={'a': 'b', 'c': 'd'})

format, which sets data to a=b;c=d. This would seem to be just as useful to people as json() because we don't want to enforce ordering on that string and it's annoying to work with. In the test i was doing i was able to use this to say:

self.assertEqual(['b'], last_request.form()['a'])

rather than parse it myself.

Launchpad Details: #LPC Jamie Lennox - 2016-10-12 12:40:09 +0000

[jamielennox]

That makes sense for libraries that might use other libraries that might add things after the fact. In that same vein, what about a multipart_form helper? This would be useful in the case of

requests.post(url, data={'a': 'b', 'c': 'd'}, files={'part-name': ('filename', open('file.txt', 'rb'))})

From a requests POV that's also a very popular feature. I recognize, however, that we don't use it often in OpenStack.

Launchpad Details: #LPC Ian Cordasco - 2016-10-12 13:20:19 +0000

[jamielennox]

Yea, I'd be keen for that. Do you have a suggestion how that would look?

There's cgi.parse_multipart which can do some of this, or somehow use cgi.FieldStorage (not sure how this works yet).

I can see a boolean is_multipart property and a multipart_form that returns a Mapping (probably just a dict) of part-name -> (filename, data, content_type)?

Launchpad Details: #LPC Jamie Lennox - 2016-10-12 23:43:27 +0000

[jamielennox]

Fix proposed to branch: master Review: https://review.openstack.org/385740

Launchpad Details: #LPC OpenStack Infra - 2016-10-13 04:20:46 +0000

[jamielennox]

That was harder than expected, the handlers around multipart forms are really bad. I've put up what i think is reasonable at [1] based on top of the other form patch and some basic rearranging. I'm going to make this bug target both a form and multipart_form helper.

[1] https://review.openstack.org/385740

Launchpad Details: #LPC Jamie Lennox - 2016-10-13 04:21:46 +0000

[jamielennox]

There's actually a much more usable version of this in the requests-toolbelt:

from requests_toolbelt.multipart import decoder

md = decoder.MultipartDecoder.from_response(resp)

That said, I realize this is something for the request object so you might do something more like

content = request.body
content_type = requests.headers.get('Content-Type')

md = decoder.MultipartDecoder(content, content_type)
for part in md.parts:
    print(part.content)
    print(part.text)
    print(part.headers)

Launchpad Details: #LPC Ian Cordasco - 2016-10-13 12:05:20 +0000

[jamielennox]

Ah, i didn't look there. I wasn't thinking about requests doing this sort of decoding so i was searching for python decoders.

So i completely agree that it's a nicer interface - however i'm not sure about taking on toolbelt as a dependency (at least it doesn't have any further dependencies) for something that hasn't been actually requested from a user yet.

Know any lighter way? Or more have any preference for how to expose the multipart_form response so it doesn't expose this underlying implementation at all?

Launchpad Details: #LPC Jamie Lennox - 2016-10-14 06:32:12 +0000

[jamielennox]

I don't particularly know a "lighter" way, short of just copying that code out of the toolbelt. I'm also not certain how you'd want to expose this frankly. I would guess something akin to a dictionary but I'm not sure how well that would work for requests-mock users.

Launchpad Details: #LPC Ian Cordasco - 2016-10-14 15:36:45 +0000

[devrelm]

~~I came upon this blog article that mentions that email.parser is in the standard python library and supports parsing multipart/form-data:~~

~~https://julien.danjou.info/handling-multipart-form-data-python/~~

~~It's a little strange to have to import from the email namespace, but it seems to work well for my purposes. I'd just love to have it abstracted away within requests-mock.~~

EDIT: Nevermind (see next comment)

[devrelm]

Nevermind, it looks like email.parse isn't actually working for me. Sorry for getting anyone's hopes up.

PR #36 looks like it's much more well-thought-out anyway.