Overcomplicated list POST in MultipartEncoder

[Guilouf]

In vanilla requests, this works well:

data = {
        'name': 'gerard',
       'type': 3,  # works fine also with int
        'many_projects': ['PROJET1', 'PROJET2']}  # works

r = requests.post('http://httpbin.org/post', data=data, files={'file': open('200mo', 'rb')},
                  headers=header)
# don't send the file if you want to see the httpbin response

But i want to stream the data, so from want i understood i have to use MultipartEncoder:

  data = {
       'name': 'gerard',
       'type': '3',  # int does not work
       # 'many_projects': ("PROJET1", "PROJET2"),  # sended as a file ,by MultipartEncoder, not working
       'many_projects': json.dumps(['PROJET1', 'PROJET2']),  # >> [\"PROJET1\", \"PROJET2\"]", not understood as a list by the server
       'file': ('filename', open('200mo', 'rb')),
        }
m = MultipartEncoder(fields=data)
header['Content-Type'] = m.content_type
r = requests.post('http://httpbin.org/post', data=m, headers=header)

So the solution from #160 didn't work for me, because the need is to send the same key with different values.

I finally found a way in #190 , but its quite ugly to transform a dictionary in to a list of tuples:

data = [
    ('name', 'gerard'),
    ('type', '3'),
    ('many_projects', 'PROJET1'),
    ('many_projects', 'PROJET2'),
    ('file', ('filename_is', open('200mo', 'rb'))),
]

My question, why it's quite simple and "magic" with request, and so complicated with request-toolbelt, in case we need the ability to stream ? Because i want sometimes to stream big files, i have to adapt and make wrapper for all the code to adapt for the low level behavior of toolbelt.

Can be a default behavior that transform list values into duplicated keys ?

[sigmavirus24]

The behavior difference is rooted in two things:

1. This was written before that behavior existed in requests. It’s underspecified behavior as is and unnecessarily magical.

2. The requests-toolbelt behavior is compatible with Requests.

3. Many people want to preserve order in these forms. For example, S3 requires exact ordering of forms. Nudging people into using lists with triples is overall more hygienic and good practice.

Finally, no I don’t think the toolbelt should mimic Requests in this instance. But I’m open to hearing others’ opinions too.

—- Written on my phone with my typo-happy thumbs. Please excuse my brevity

[Guilouf]

Thank you for leaving a reply, i understand more your point of view. Besides, both solutions, (dict with list inside and list of tuples) could be implemented, like it is in vanilla request, both works. Thus, we could use list of tuple or dict if we want to preserve the order or not, or even use an OrderedDict.

My concern is a kind of lack of consistency between streaming and non streaming multipart, even if i can resolve this by using only list of tuples or use the MultipartEncoder.to_string() to switch between stream / no stream.

[sigmavirus24]

Requests will happily accept a list of tuples. If it doesn't have the same behaviour as the Encoder, that sounds like a bug and a regression in behaviour.

[Guilouf]

Requests will happily accept a list of tuples.

Yea, that's what i said, request will accept a list of tuples or a dict with list inside, and both will work. Toolbelt only accept a list of tuple in the case of multiple values for one key, that's where my concern is, because if my data is in a dict, it will work with request for non streamed multipart data, but i will have to transform it into a list of tuple to use MulipartEncoder and thus stream multipart data.

Btw, the solution for me if i want consistency between streaming / non streaming is to use only list of tuples.

[sigmavirus24]

Btw, the solution for me if i want consistency between streaming / non streaming is to use only list of tuples.

Right. I understand the appeal of having the dictionary there. It makes sense. Like I said, I think pushing people towards ordered structures, however, makes far more sense as an issue of hygiene.