Too much load on bgp.tools

[benjojo]

For the past few months bgp.tools has been gaining more and more traffic table.jsonl from a mysterious chrome user agent, from a large range of what are suspected to be VPN IP ranges.

This user agent:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36

Is against the API/Scripting recommendations of the site:

You must a HTTP User-Agent to identify yourself on these requests. Default user agents are not supported as we have no contact method to reach you if something goes wrong.

Systems who use a default or generic user agents to fetch bgp.tools data constantly may be blocked.

The preferred user agent format is:

acmeco bgp.tools - [email protected]

The current request rate from this user-agent (that has been blocked for at least the last 60 days) has reached over 5 RPS.

That sounds small, however that makes a huge bandwidth impact when the file it is fetching is 50MiB, meaning to serve that the website would be pumping out 2gbit/s of pure jsonl file to serve this workload. At the cheapest possible carrier rates that would cost at least $100/m in bandwidth to serve.

Had I been serving this out from AWS Cloudfront this would cost me $29,450 a month.

Anyway, I investigated this by looking up bgp.tools on virustotal and found that amass is used in what look like a large number of payloads. And upon investigation, there is code that behaves as I am witnessing.

It seems reasonable to assume there is a large malware campaign using amass right now due to the sheer % of these requests coming out of VPN ranges

https://github.com/OWASP/Amass/blob/master/resources/scripts/api/bgptools.ads added on https://github.com/OWASP/Amass/commit/5abed3b1ca54a4a88cab96e3e69ae06e865d3da8 by @caffix appears to be driving this load. And that Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36 appears to be the default user agent of amass.

My requests are:

A) Please do not use the default user agent here. This has taken me significant time to discover. Please consider using owasp amass <version> for easy identification. B) Please do not fetch table.jsonl if you think you are only going to be making 1 or 2 lookups. Please use the whois server instead.

---

bgp.tools plans to display a banner warning of compromise for the IPs that are hitting this. Since it seems that these requests look highly likely to be of malware aligned amass installs