Rustification - Web Library

[TheTechromancer]

Requirements

• An async client with the following functions:
  • .request() - a one-off request similar to requests
  • .request_batch() - a dedicated pool that, given a list of URLS to visit, yields responses as they happen
  • .request_custom_batch() - similar to request_batch() but each URL can have custom args/kwargs
  • .download() - Download to a local file

Ideally:

• Client is configurable via rate limits, proxy, custom headers, debug logging, etc.
• Benchmarked faster than httpx (especially for request_batch())

TODO

• [ ] Research rust web libraries (benchmark performance)
  • Must be async, preferably low enough level to do nasty things
  • Top candidate so far: https://github.com/hyperium/hyper

Basic prototype here: https://github.com/blacklanternsecurity/rustweb

We could learn a lot from go-retryablehttp, which is the library at the bottom of ProjectDiscovery's tools, including httpx.

• https://github.com/hashicorp/go-retryablehttp

[liquidsec]

merging with https://github.com/blacklanternsecurity/bbot/issues/1569