httpx icon indicating copy to clipboard operation
httpx copied to clipboard

When the URL contains request parameters and the `params` parameter is set, the request parameters in the URL will disappear unexpectedly.

Open 2513502304 opened this issue 4 months ago • 2 comments

The starting point for issues should usually be a discussion...

https://github.com/encode/httpx/discussions

Possible bugs may be raised as a "Potential Issue" discussion, feature requests may be raised as an "Ideas" discussion. We can then determine if the discussion needs to be escalated into an "Issue" or not.

This will help us ensure that the "Issues" list properly reflects ongoing or needed work on the project.


  • [ ] Initially raised as discussion #...

2513502304 avatar Aug 05 '25 14:08 2513502304

As shown in the following Python code, my expected output should be to reasonably concatenate the request parameters in params with those in the url. requests handled this part well, but httpx gave the following unexpected output:

import requests
import httpx

url = 'https://safebooru.org/index.php?page=post&s=list'
params = {
    'pid': 0,
    'tags': 'k-on!',
}

response = requests.get(url=url, params=params)
print(response.request.url)  # As expected: https://safebooru.org/index.php?page=post&s=list&pid=0&tags=k-on%21

response = httpx.get(url=url, params=params)
print(response.request.url)  # Unexpectedly, the request parameters in the URL were overridden by params: https://safebooru.org/index.php?pid=0&tags=k-on%21

My httpx version is 0.28.1

2513502304 avatar Aug 05 '25 14:08 2513502304

If you want to avoid params getting dropped/overwritten when a URL already contains a query, you can pre-merge the query yourself with qs_codec (my Python port of the Node.js qs library) and hand httpx a fully-formed URL. This also gives you proper nested param handling (a[b][0]=...) and configurable encoding (spaces as %20 vs +, etc).

Safely merge existing URL query with new params

import httpx
from urllib.parse import urlsplit, urlunsplit
import qs_codec as qs

base_url = "https://api.example.com/search?a=b&c=d"  # already has query
new_params = {
    "c": "D",               # will co-exist or override per your choice below
    "e": "f",
    "tags": ["x", "y"],     # lists -> multiple entries or bracketed, as configured
}

parts = urlsplit(base_url)

# Decode existing query string into a nested dict (robust parsing, preserves duplicates)
existing = query_string.decode(parts.query)

# Merge how you want (examples):
# a) combine duplicates into a list
for k, v in new_params.items():
    if k in existing:
        # promote to list and append
        existing[k] = ([existing[k]] if not isinstance(existing[k], list) else existing[k]) + (v if isinstance(v, list) else [v])
    else:
        existing[k] = v

# b) OR choose "last-wins" by just updating:
# existing.update(new_params)

# Re-encode exactly once (RFC 3986 = spaces as %20; switch to RFC1738 for '+')
query_string = qs.encode(existing, qs.EncodeOptions(format=qs.Format.RFC3986))

final_url = urlunsplit((parts.scheme, parts.netloc, parts.path, query_string, parts.fragment))
resp = httpx.get(final_url)

Nested query parameters (qs-style)

If the API expects bracketed or indexed keys (a common pain point), qs_codec handles it:

# Build nested structure directly in Python
payload = {
    "search": {"term": "billing", "filters": [{"field": "status", "op": "eq", "value": "open"}]},
    "page": 2
}

# Encodes to: search[term]=billing&search[filters][0][field]=status&...&page=2
query_string = qs.encode(payload)  # percent-encoded by default
url = f"https://api.example.com/tickets?{query_string}"
resp = httpx.get(url)

Encoding edge cases (spaces, %, etc.)

qs_codec lets you pick encoding style and avoids double-encoding:

  • RFC 3986 (default) → space = %20
  • RFC 1738 → space = +
qs.encode({"q": "foo bar"}, qs.EncodeOptions(format=qs.Format.RFC3986))  # q=foo%20bar
qs.encode({"q": "foo bar"}, qs.EncodeOptions(format=qs.Format.RFC1738))  # q=foo+bar

This approach works today without changes to httpx: you bypass the internal params merge (the source of surprises here) and send an already-correct query string.

techouse avatar Aug 28 '25 21:08 techouse