`httpx` crashes on services generating endless stream of data

[ocervell]

httpx version:

v1.6.8

Current Behavior:

httpx hangs and the machine becomes OOM when hitting a service that generates an endless stream of data (memory usage keeps growing until crash).

In my case, it is a CHARGEN UDP service running on my internet box (provider: freebox).

nmap detection:

nmap -oX /home/osboxes/.secator/reports/default/tasks/18626/.outputs/nmap.xml <REDACTED> -p 8095 --script vulners -Pn -sV -sT
Starting Nmap 7.93 ( https://nmap.org ) at 2024-10-21 19:15 EDT
Nmap scan report for [REDACTED]
Host is up (0.00053s latency).
Other addresses for X (not scanned): <REDACTED>
rDNS record for <REDACTED>
PORT     STATE SERVICE VERSION
8095/tcp open  chargen xinetd chargen
Service Info: OS: Unix
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 5.27 seconds

httpx command:

httpx -u <MYHOST>:8095  # hangs, running with -debug OOMs the machine because of the endless stream

Note that adding a --timeout 1 --retries 0 will fix the issue, as well as restricting the max body size to read (-rstr 2000) but it would be preferable to fix without as it hinders the actual issue.

This is problematic in automatic workflows as any service running a char generator or yielding an endless stream of data will crash the machine the workflow is running on.

Expected Behavior:

httpx should:

• not store the entire response in memory, as the used memory grow with the response body size, it should stream the data directly to a file if possible (it's preferable to use a bit more disk space than memory when running in resource-constrained environments) OR (harder / possibly not feasible):
• detect that the stream keeps growing and stop after a while.

Steps To Reproduce:

• Run a CHARGEN service: https://www.ncsc.gov.ie/emailsfrom/Shadowserver/DoS/Chargen/
• Hit it with httpx: httpx -u <MYHOST>:8095

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions!

[smallseacreature]

seeing if I can reproduce this, will keep you updated!