httpx icon indicating copy to clipboard operation
httpx copied to clipboard

Rate Limit Minute (-rlm) behavior does not distribute requests evenly

Open swdbo opened this issue 11 months ago • 0 comments

httpx version:

v1.6.0

Current Behavior:

When using the -rlm (rate limit per minute) argument, the tool sends the specified amount of requests all at once instead of distributing them evenly across the minute. This behavior is counterintuitive, as it leads to a burst of traffic followed by a period of inactivity, rather than spreading the requests out to avoid overwhelming target servers.

Expected Behavior:

The expected behavior for the -rlm argument is to distribute the specified number of requests evenly across the minute. For instance, if -rlm 10 is specified, one would expect a request to be sent every 6 seconds, thereby evenly pacing the load on the target server(s) and adhering more closely to a "rate limit."

Steps To Reproduce:

  1. Prepare a urls.txt file with multiple target URLs.
  2. Run httpx with verbose logging and the -rlm argument set to 10, like so: cat urls.txt | httpx -v -rlm 10
  3. Observe the output and timing of requests; all 10 requests are made at the same time, not spaced out at 1 request every 6 seconds as expected.

Anything else:

This unexpected behavior could lead to potential flooding of target websites, which is especially concerning in scenarios where careful rate limiting is necessary to comply with target server policies or to avoid unintentional Denial-of-Service conditions. An adjustment to ensure requests are distributed evenly throughout the specified time frame would greatly enhance the utility and reliability of the -rlm feature.

swdbo avatar Mar 07 '24 16:03 swdbo