aiohttp icon indicating copy to clipboard operation
aiohttp copied to clipboard

Documentation misleading on query string encoding

Open bsolomon1124 opened this issue 6 years ago • 1 comments

Long story short

The docs in Passing Parameters in URLs is misleading and the behavior is essentially the opposite of what is stated:

You can also pass str content as param, but beware – content is not encoded by library. Note that + is not encoded.

Expected behaviour

Passing the fully encoded querystring params=url=https%3A%2F%2Fwww.katherinetimes.com.au%2Fstory%2F6005621%2Fwife-of-ex-nissan-boss-ghosn-leaves-japan%2F%3Fsrc%3Drss&api_key=XXXX should not requote the querystring. I'm assuming that's what "content is not encoded by library" is supposed to mean.

What is more accurate is that passing a URL with encoded=True to the url of request() is the only way to get this behavior.

Actual behaviour

From client_reqrep.py:

            q = MultiDict(url.query)
            url2 = url.with_query(params)
            q.extend(url2.query)
            url = url.with_query(q)

Okay, let's try to interpret the above:

>>> import urllib.parse
>>> from yarl import URL
>>> 
>>> base = "https://api.example.com" 
>>> params = {
...     "url": "https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src=rss",
...     "api_key": "XXXX",
... }
>>> u = URL(base).with_query(urllib.parse.urlencode(params))
>>> u
URL('https://api.example.com/?url=https://www.katherinetimes.com.au/story/6005621/wife-of-ex-nissan-boss-ghosn-leaves-japan/?src%3Drss&api_key=XXXX')
>>> urllib.parse.urlencode(params)
'url=https%3A%2F%2Fwww.katherinetimes.com.au%2Fstory%2F6005621%2Fwife-of-ex-nissan-boss-ghosn-leaves-japan%2F%3Fsrc%3Drss&api_key=XXXX'

So the query string in u is requoted! (encoded=True always in .with_query().)

What are the docs trying to say here? Am I misinterpreting the phrase "content is not encoded by library"? How so?

Steps to reproduce

See above.

Your environment

aiohttp client 3.5.4, Python 3.7.2, yarl 1.3.0.

bsolomon1124 avatar Apr 11 '19 21:04 bsolomon1124

I can confirm that the issue still exists and there is no way to prevent re-encoding of params

This is crucial for OData services some of which "prefer" when whitespaces are percentage encoded as %20 instead of being replaced by +

grenik avatar Oct 30 '25 20:10 grenik