blacklanternsecurity
Optimize time to create event
Creating an event object in BBOT takes a relatively long time, approximately .0003 seconds. While this doesn't sound like much, it adds up very quickly and is the reason why when you have a big target list, BBOT takes so long to start.
We should benchmark event creation and figure out which part of it is hogging the most CPU.
Some progress has been made on this, by creating a lightweight, intermediate EventSeed class. But most of the performance hit is coming from calls to misc helpers like smart_encode_punycode() and split_host_port(), which contain lots of icky regex and string comparisons.
We would see the most benefit by converting these low-level helpers (and honestly, most of our builtin DNS/IP/URL regexes, validators, etc.) to Rust.
For funzies, made a benchmark to compare IDNA / punycode encoding in rust vs pyo3 vs python.
Benchmark Results
Encode
-
Raw Rust: 0.000004 s (4 microseconds)
-
Rust (Python): 0.036057 s (36 milliseconds)
-
Python IDNA: 0.261474 s (261 milliseconds)
-
Rust (Python) overhead: 0.036057 / 0.000004 ≈ 9,014x slower than raw Rust
-
Python IDNA overhead: 0.261474 / 0.000004 ≈ 65,368x slower than raw Rust
-
Rust (Python) vs Python IDNA: 0.261474 / 0.036057 ≈ 7.25x slower (so Rust in Python is still much faster than pure Python)
Decode
-
Raw Rust: 0.000018 s (18 microseconds)
-
Rust (Python): 0.141961 s (142 milliseconds)
-
Python IDNA: 0.245140 s (245 milliseconds)
-
Rust (Python) overhead: 0.141961 / 0.000018 ≈ 7,887x slower than raw Rust
-
Python IDNA overhead: 0.245140 / 0.000018 ≈ 13,619x slower than raw Rust
-
Rust (Python) vs Python IDNA: 0.245140 / 0.141961 ≈ 1.73x slower (Rust in Python is still faster than pure Python)
Conclusion
- Raw Rust is orders of magnitude faster than both Rust called from Python and pure Python IDNA.
- Calling Rust from Python (via PyO3) introduces a large overhead (about 8,000–9,000x slower than pure Rust), but it is still significantly faster than pure Python (about 1.7–7x faster, depending on the operation).
- Most of the time in the Python benchmarks is spent in the FFI boundary and Python call overhead, not in the Rust code itself.
- If you need absolute maximum performance, use Rust natively.
- If you need Python integration, using Rust via PyO3 is still a big win over pure Python, but you lose several orders of magnitude in speed compared to native Rust.