Do not raise exception for invalid redirect URLs

[devl00p]

Related discussion: https://github.com/encode/httpx/discussions/3179

May also be related to several discussions tied to InvalidURL exceptions

Description:

Currently, httpx raises an httpx.InvalidURL exception when a response contains a Location header with a URL that is not considered valid, such as one with a non-HTTP scheme like data:.

This behavior is not ideal, as an HTTP response with a malformed Location header is still a valid HTTP response. The client should not crash in this case, but rather allow the user to inspect the response, including the problematic Location header.

How to Reproduce the Issue

The following code demonstrates the original issue. The target URL returns a 302 redirect where the Location header contains a data: URI.

import httpx

# This URL returns a 302 redirect with a 'data:' URL in the Location header.
url = "http://angular.testsparker.com/test.php?r=data%3A%3Bbase64%2CPD9waHAgZWNobyAndzRwMXQxJywnX2V2YWwnOyA%2FPg%3D%3D"

# Before this fix, the following line would raise httpx.InvalidURL
response = httpx.get(url, follow_redirects=True)

# With this fix, the request completes successfully.
# The redirect is not followed, and the original response is returned.
assert response.status_code == 302
assert response.next_request is None

The Fix

This MR modifies the redirect handling logic. Now, if a redirect URL is invalid, httpx will stop following redirects and return the original redirect response. The response.next_request property will be set to None to indicate that no valid redirect request could be constructed.

The tests for invalid redirects have been updated to reflect this new, more robust behavior.

Use Case: Web Vulnerability Scanning

This change is motivated by the use of httpx within the Wapiti web vulnerability scanner.

A core function of such scanners is to send a wide variety of payloads, some of which are intentionally malformed, to probe for security flaws. This can cause servers to respond in unexpected or non-standard ways. For instance, a server might reply with a redirect containing an invalid Location header, as demonstrated in this MR.

In this scenario, the HTTP client must be robust and fault-tolerant. Instead of crashing with an InvalidURL exception, it should gracefully handle the response and allow the calling application (the scanner) to inspect it. This change ensures httpx behaves more predictably and resiliently, making it a more reliable choice for security tooling and other applications that interact with potentially unpredictable endpoints.

[mimi89999]

I'm not convinced by this change. While I agree that if follow_redirects is set to False no errors should be raised, I think that then that parameter is set to True, an error should be raised. After all, we asked the client to follow the redirect and it could not do that. It should not fail silently.

I really like the approach of aiohttp https://docs.aiohttp.org/en/stable/_modules/aiohttp/client_exceptions.html#InvalidUrlRedirectClientError where the exception InvalidUrlRedirectClientError(InvalidUrlClientError, RedirectClientError) is both an invalid URL error and a redirect error.