BBOT 3.0 - blazed_elijah

[TheTechromancer]

Summary

BBOT 3.0 "blazed_elijah" contains changes needed to store BBOT data in a persistent database. The idea is to release it alongside BBOT server, a tiny CLI-only database. This will be paired with a series of blog posts showing how BBOT server can be used on the command line to script out bug bounty hunting, threat intel, and ASM (i.e. running scheduled scans, exporting to CSV, diffing results over time, etc.).

Together, BBOT 3.0 and BBOT server will give us a solid foundation to build a bunch of other useful tooling, like asset inventory. Sometime in the future, it may also be useful to frontend.

Breaking changes

1. .data and .data_json event fields

The main breaking change in BBOT 3.0 is that the name of the .data field is different based on whether it's a str or dict.

• .data: string
• .data_json: dictionary

The siem_friendly option has been removed, since BBOT data is now SIEM-friendly by default.

2. Changes to vulnerabilities

The VULNERABILITY event type has been removed in favor of FINDING, which now has several improvements:

• A name field which holds a generic description common to all findings of the same type. This makes it easier to collapse and categorize them.
• A confidence field
• A severity field

Features

• https://github.com/blacklanternsecurity/bbot/pull/1992
• https://github.com/blacklanternsecurity/bbot/pull/2010
• https://github.com/blacklanternsecurity/bbot/pull/2013
• https://github.com/blacklanternsecurity/bbot/pull/2011
• https://github.com/blacklanternsecurity/bbot/pull/2015
• https://github.com/blacklanternsecurity/bbot/pull/2017

Potential changes

• https://github.com/blacklanternsecurity/bbot/issues/1592

[codecov[bot]]

Codecov Report

:x: Patch coverage is 93.77457% with 95 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 92%. Comparing base (c977c26) to head (95b5dd4). :warning: Report is 18 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/constants.py	71%	11 Missing :warning:
bbot/scanner/scanner.py	87%	11 Missing :warning:
bbot/modules/base.py	68%	10 Missing :warning:
bbot/modules/output/nats.py	80%	7 Missing :warning:
bbot/core/event/base.py	88%	6 Missing :warning:
bbot/models/pydantic.py	94%	6 Missing :warning:
bbot/modules/output/mongo.py	90%	6 Missing :warning:
bbot/modules/internal/excavate.py	80%	5 Missing :warning:
bbot/core/config/logger.py	20%	4 Missing :warning:
bbot/modules/output/zeromq.py	88%	4 Missing :warning:
... and 11 more

Additional details and impacted files

@@          Coverage Diff           @@
##             dev   #2007    +/-   ##
======================================
+ Coverage     92%     92%    +1%     
======================================
  Files        411     428    +17     
  Lines      34044   34825   +781     
======================================
+ Hits       31064   31799   +735     
- Misses      2980    3026    +46     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions[bot]]

📊 Performance Benchmark Report

Comparing dev (baseline) vs 3.0 (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	4.26ms	4.33ms	+1.6% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	18.59ms	17.41ms	-6.3% ⚪	✅
Large Closest Match Lookup	355.67ms	361.51ms	+1.6% ⚪	✅
Realistic Closest Match Workload	195.66ms	192.80ms	-1.5% ⚪	✅
Event Validation Full Scan Startup Small Batch	463.60ms	494.19ms	+6.6% ⚪	✅
Event Validation Full Scan Startup Large Batch	828.56ms	901.53ms	+8.8% ⚪	✅
Make Event Autodetection Small	32.41ms	30.71ms	-5.2% ⚪	✅
Make Event Autodetection Large	325.41ms	320.41ms	-1.5% ⚪	✅
Make Event Explicit Types	14.19ms	14.16ms	-0.2% ⚪	✅
Excavate Single Thread Small	4.057s	4.101s	+1.1% ⚪	✅
Excavate Single Thread Large	9.393s	9.560s	+1.8% ⚪	✅
Excavate Parallel Tasks Small	4.181s	4.149s	-0.8% ⚪	✅
Excavate Parallel Tasks Large	7.077s	7.091s	+0.2% ⚪	✅
Is Ip Performance	3.16ms	3.18ms	+0.7% ⚪	✅
Make Ip Type Performance	11.40ms	11.63ms	+2.0% ⚪	✅
Mixed Ip Operations	4.48ms	4.53ms	+1.2% ⚪	✅
Typical Queue Shuffle	62.54µs	62.67µs	+0.2% ⚪	✅
Priority Queue Shuffle	707.64µs	705.14µs	-0.4% ⚪	✅

🎯 Performance Summary

✅ No significant performance changes detected (all changes <10%)

---

🐍 Python Version 3.11.14

[liquidsec]

@TheTechromancer we should now add to the list of breaking changes, the removal of vulnerability event type and the addition of severity and confidence as attributes on finding types