urequests: miscalculation of Content-Length when dealling with unicode data

[7th-heaven]

code in urequests ver 0.5.1: s.write(b"Content-Length: %d\r\n" % len(data))

"len" calcs a single unicode as 1 length. to correct it. encode the data before calcs the length as: s.write(b"Content-Length: %d\r\n" % len(data.encode(''))) plz fix it.

[pfalcon]

Perhaps, you should call .encode() on your side instead?

[7th-heaven]

Yup, it's works correctly in micorpython 1.9.3 on nodemcu with eps8266. And ascii data works well too.

Before I fix this, There's no problem when dealling with ascii datas. But, as I post the unicode data to WebHook on the server via https.The server replies 'illegal parameters'.

[pfalcon]

If it wasn't clear, you should call .encode() method in your application, and pass bytes as "data" param to urequests. You can see this requirement in the upstream requests module too: http://docs.python-requests.org/en/master/api/#requests.request

(This issue can remain open. If there will be a dozen more similar reports from users, the issue can be reconsidered.)

[7th-heaven]

def post_data(data):
    url='WEBHOOKURL'

    jsonmsg = {
        "msgtype": "text",
        "text": {
            "content":   data
        },
        "at": {
        "atMobiles": 
          [],
        "isAtAll": False
              }
        }     
    jsonheader = {
        'Content-Type': 'application/json'
    }
    r = urequests.post(url, data=json.dumps(jsonmsg), headers=jsonheader)
    return r

This is the part of my app. This function posts unicode text to server, and then the server send the unicode text to the clients. the unicode text can not call .encode() before POST.

For exsamples:

1. I called post_data('测试') , in requests-lib of python3 on PC, it's worked succesfully. BUT in urequests-lib of micropython on nodemcu with esp8266, IT DO NOT WORKS . the server replies 'illegal parameters' to me, and clients received nothing. It's casued by wrong Content-Length.

2. I called post_data(str('测试').encode('')) , the server replies 'OK' to me, clients received '测è¯' Oh, it's works. but gibberish recevied.

I'v tried data = None, headers = None, json = jsonmsg. The resluts are the same as exsamples. Is the wrong way I tried?

[kenjican]

@7th-heaven I encountered same issue,, this is my solution: data should be encode as bytes and pass headers parameter: res = urequests.post(url, data=(ujson.dumps(dd['msg'])).encode(), headers={'Content-Type':'application/json'})

It looks like DingTalk's API....I've tested and proven that works

[projectgus]

This is very similar to #746 which is a report about the same behaviour with umqtt.

Have the same suggestion here, that silently truncating utf-8 arguments is probably not a good choice - it should either explicitly fail, or attempt to encode even if this does cost some RAM (and the documentation can recommend passing bytes to save RAM).