Portscan CDN Detection

[TheTechromancer]

This PR adds the option to ignore certain open ports if their host belongs to a CDN.

To use it, you set the option modules.portscan.allowed_cdn_ports:

bbot -t evilcorp.com -m portscan -c modules.portscan.allowed_cdn_ports=80,443

Addresses https://github.com/blacklanternsecurity/bbot/issues/1967.

Congratulations @Sh4d0wHunt3rX, you win.

---

Exclude CDNs from Port Scan

If you want to exclude CDNs (e.g. Cloudflare) from port scanning, you can set the allowed_cdn_ports config option in the portscan module. For example, to allow only port 80 (HTTP) and 443 (HTTPS), you can do the following:

bbot -t evilcorp.com -m portscan -c modules.portscan.allowed_cdn_ports=80,443

By default, if you set allowed_cdn_ports, it will skip only providers marked as CDNs. If you want to skip cloud providers as well, you can set cdn_tags:

bbot -t evilcorp.com -m portscan -c modules.portscan.allowed_cdn_ports=80,443 modules.portscan.cdn_tags=cdn,cloud

...or via a preset:

modules:
  - portscan

config:
  modules:
    portscan:
      allowed_cdn_ports: 80,443
      cdn_tags: cdn,cloud

bbot -t evilcorp.com -p skip_cdns.yml

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 55.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 93%. Comparing base (3945fd1) to head (1286a94). Report is 108 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/modules/portscan.py	44%	9 Missing :warning:

Additional details and impacted files

@@          Coverage Diff           @@
##             dev   #1970    +/-   ##
======================================
+ Coverage     93%     93%    +1%     
======================================
  Files        365     370     +5     
  Lines      27916   28326   +410     
======================================
+ Hits       25723   26120   +397     
- Misses      2193    2206    +13     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[TheTechromancer]

@Sh4d0wHunt3rX @Luoooio, should this take into consideration all cloud providers (i.e. Google, Amazon) or only clear-cut CDNs like cloudflare?

There is a tradeoff, as including these will speed up scans but also increase the chance of missing stuff.

[Sh4d0wHunt3rX]

@TheTechromancer My approach was using my friend's tool: https://github.com/ImAyrix/cut-cdn

Then exclude any IP that belongs to these CDNs for port scanning.

[Luoooio]

@TheTechromancer In the current code, I noticed that you use cloudcheck to determine if it’s a cloud provider or CDN. I reviewed the code logic, and it adopts a fairly rigorous method for detection, making it quite accurate for known providers. I used a similar approach in my own tools but added CNAME checks and maintained a mapping of CDNs and CNAMEs (though it has since become outdated). Here's a small tip to share: I typically add two additional checks on top of this: 1. Checking if the CNAME contains a CDN-related string 2. If an IP has more than 10 open ports, I consider it not worth scanning further. This helps me cut down a lot of noise.

As for your question, my suggestion would be to add an option allowing users to decide for themselves, as different users have different needs.

[TheTechromancer]

Thanks guys; yeah right now we take both CNAMEs and IPs into consideration, and those lists are auto-updated daily. But for bigger cloud providers like Amazon, we don't distinguish between their CDN and the rest of their servers. This shouldn't be too hard to do, though. I think cut-cdn has some good sources for that.