crowdsecurity
feat: add parser to allowlist for AWS global service IPs
Hello,
Unless I'm mistaken, this effectively allowlist all the AWS IPs ? (including EC2).
While allowing cloudfront is probably a good idea, allowing any EC2 IP seems kinda dangerous: even if AWS responds to abuse, some of their servers have malicious behaviours (AWS is in the top 5 AS in the community blocklist).
From a performance perspective, I'd expect this parser to be extremely slow: JsonExtract will reparse the JSON each time it is called, and there are about 15k ranges in the JSON file:
- Each log line will trigger parsing of a not-so-small JSON file
- In the worst case, the IP in the log line will be checked against 15k ranges, which will take some time.
Ideally, this file should be pre-processed and have a single range per line, which would allow to just use File and any without any parsing.
This is actually part of a larger subject: currently we have some scenarios that fetch datafiles from 3rd parties (eg, cloudflare ip ranges), I'd like to centralize everything in a repo we own (for example, https://github.com/crowdsecurity/sec-lists/) and have a CI job update everything automatically. This would allow for easy pre-processing if needed (and ideally, make sure crowdsec can use the same whitelists we use server-side for the consensus)
Additionally, this kind of resource-intensive match should be instead performed in a postoverflow: they will only be checked after a scenario was triggered, which should happen much less often.
