feat: retry on 429

[jjhwan-h]

Summary

This PR introduces a round-robin retry mechanism for HTTP 429 (Too Many Requests) responses. Users can control the number of retries and the delay between them using two new flags:

--retry-rounds

--retry-delay

Changes

Options

Added RetryRounds (int) → number of retries for 429 responses

Added RetryDelay (int) → delay between retries (in ms)

Added validation to ensure retries and delay values are valid

Runner

Integrated a new retryLoop with drainedCh synchronization

Introduced retryJob struct for retry queue management

On 429 responses, jobs are pushed into the retryCh for delayed execution

Used atomic.Int64 to track remaining retries and close drainedCh when done

Tests

Added TestRunner_Process_And_RetryLoop

srv1: Always returns 429 → retries exhausted, expected 3×429, 0×200

srv2: Returns 2×429 then 200 → expected 2×429, 1×200

Verified drain and retry logic

Example

httpx -l urls.txt --retry-rounds 2 --retry-delay 500

A request returning 429 will be retried up to 2 times

Each retry happens after 500ms delay

If retries are exhausted, the request is considered failed

Related Issue

#1078

Summary by CodeRabbit

• New Features

  • Automatic retries for HTTP 429 (rate-limited) responses with configurable rounds and delay.
  • New CLI flags: --retry-rounds and --retry-delay.
  • Run completion now waits for all retry attempts to finish.
  • Validation: when retry rounds > 0, retry delay must be > 0.

• Tests

  • Added a concurrency test verifying retry behavior and handling of rate-limited responses.

[coderabbitai[bot]]

Walkthrough

Adds configurable HTTP 429 retry support: new Options fields and CLI flags with validation; introduces a retryJob type, retry channel, and retryLoop with scheduling and atomic draining; propagates retry channel through processing (including nested probes); updates Runner method signatures and adds tests.

Changes

Cohort / File(s)	Summary
Options API and CLI flags runner/options.go	Adds RetryRounds and RetryDelay fields to Options, registers --retry-rounds and --retry-delay CLI flags, and validates RetryDelay > 0 when RetryRounds > 0.
Retry mechanism core runner/runner.go, runner/...	Adds retryJob type (runner/types.go), introduces retryCh channel, implements retryLoop with scheduling and atomic in-flight tracking, enqueues retries on HTTP 429 up to RetryRounds with RetryDelay, and threads retryCh through Process/process and nested probe calls.
Tests runner/runner_test.go	Adds TestRunner_Process_And_RetryLoop to exercise 429->retry behavior across servers; includes a minor whitespace alignment tweak in another test.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor User as RunEnumeration
  participant W as Worker/process
  participant RL as RetryLoop
  participant S as HTTP Server
  participant O as Output

  rect rgba(230,245,255,0.6)
    User->>W: process(target, retryCh)
    W->>S: analyze request
    S-->>W: response (200 / 429)
    alt 429 and attempts < RetryRounds
      W->>RL: enqueue retryJob (when = now + RetryDelay)
    end
    W-->>O: emit result
  end

  rect rgba(240,255,240,0.6)
    loop until drained
      RL-->>RL: wait until(job.when)
      RL->>S: analyze retry request
      S-->>RL: response
      RL-->>O: emit retry result
      alt 429 and attempts < RetryRounds
        RL->>RL: re-enqueue retryJob
      end
    end
  end

  note over User,RL: Drained signaled when initial + retry jobs complete

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

I nibbled code paths, hop by hop,
Queued tiny retries when servers said "stop".
A delay and a round, then another try—
Counting attempts as the packets fly.
When channels are empty and burrows clear,
I twitch my nose and thunder forth a cheer. 🐇✨

[!TIP]

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches

• [ ] 📝 Generate Docstrings

🧪 Generate unit tests

• [ ] Create PR with unit tests
• [ ] Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[Mzack9999]

Nice one! I was thinking about something similar with automatic detection of rate limit from RL headers combined with https://github.com/projectdiscovery/ratelimit/tree/feat-per-key-limit to make the tool compliant with them without slowness. Just curious, were you testing many different paths towards the same target in order to hit 429? Generally httpx use case is to send few requests to a very large number of targets.

[jjhwan-h]

Thanks for the feedback!

In my testing I was indeed sending many different paths to the same target, which resulted in 429 responses. However, in real-world environments with CDNs and WAFs, 429s can also occur sporadically across multiple targets, even with only a few requests per host.

The --retry-rounds flag is therefore intended as an optional safeguard to preserve result accuracy in such cases. It does not alter the default behavior and can be enabled only when needed.

if you feel additional tests or adjustments to the implementation would make this more useful or easier to maintain, I’d be happy to revise the PR accordingly!