crowdsec
crowdsec copied to clipboard
crowdsecurity
getDecisionStream does not return custom scope decisions by default
What happened?
Without the scopes parameter set, getDecisions returns Ip, Range, and custom scopes (UserAgent here) decisions:
curl -sS -H "X-Api-Key: 9E1uMkxZzSDYQZU/Qbm9Og" -k 'https://127.0.0.1:8080/v1/decisions?origins=cscli%2Ccrowdsec' | jq .
[
{
"duration": "3h50m28.543341271s",
"id": 6177362,
"origin": "cscli",
"scenario": "manual 'ban' from ''",
"scope": "UserAgent",
"type": "ban",
"value": "crowdsec"
},
{
"duration": "3h50m42.543340911s",
"id": 6177363,
"origin": "cscli",
"scenario": "manual 'ban' from ''",
"scope": "Ip",
"type": "ban",
"value": "1.2.3.4"
},
{
"duration": "3h59m54.543340425s",
"id": 6177364,
"origin": "cscli",
"scenario": "manual 'ban' from ''",
"scope": "Range",
"type": "ban",
"value": "1.2.3.0/24"
}
]
However, by default (i.e. without scopes being set) getDecisionsStream does not return the custom scopes decisions, acting as in an scopes=Ip,Range filter was set:
curl -sS -H "X-Api-Key: 9E1uMkxZzSDYQZU/Qbm9Og" -k 'https://127.0.0.1:8080/v1/decisions/stream?startup=true&origins=cscli%2Ccrowdsec' | jq .
{
"deleted": null,
"new": [
{
"duration": "3h49m55.067403477s",
"id": 6177363,
"origin": "cscli",
"scenario": "manual 'ban' from ''",
"scope": "Ip",
"type": "ban",
"uuid": "dff4470c-6e10-4b75-acf1-3a5d7dd4084c",
"value": "1.2.3.4"
},
{
"duration": "3h59m7.067401742s",
"id": 6177364,
"origin": "cscli",
"scenario": "manual 'ban' from ''",
"scope": "Range",
"type": "ban",
"uuid": "798e70b5-2d84-43aa-8cfd-8abe42b2ab71",
"value": "1.2.3.0/24"
}
]
}
This prevents me from using https://github.com/crowdsecurity/cs-custom-bouncer for acting on custom scope decisions, since those are not returned by default by the stream API that the bouncer is using.
Note that manually setting scopes to Ip,Range,UserAgent makes getDecisionsStream return those decisions:
curl -sS -H "X-Api-Key: 9E1uMkxZzSDYQZU/Qbm9Og" -k 'https://127.0.0.1:8080/v1/decisions/stream?startup=true&scopes=Ip,Range,UserAgent&origins=cscli%2Ccrowdsec' | jq .
{
"deleted": null,
"new": [
{
"duration": "3h44m57.2303507s",
"id": 6177362,
"origin": "cscli",
"scenario": "manual 'ban' from ''",
"scope": "UserAgent",
"type": "ban",
"uuid": "ae12d4ba-fedf-42d2-8e5e-20aaafeab32e",
"value": "crowdsec"
},
{
"duration": "3h45m11.230350125s",
"id": 6177363,
"origin": "cscli",
"scenario": "manual 'ban' from ''",
"scope": "Ip",
"type": "ban",
"uuid": "dff4470c-6e10-4b75-acf1-3a5d7dd4084c",
"value": "1.2.3.4"
},
{
"duration": "3h54m23.230349652s",
"id": 6177364,
"origin": "cscli",
"scenario": "manual 'ban' from ''",
"scope": "Range",
"type": "ban",
"uuid": "798e70b5-2d84-43aa-8cfd-8abe42b2ab71",
"value": "1.2.3.0/24"
}
]
}
What did you expect to happen?
I would expect all scopes to be returned by default if scopes is not set, matching how the non-stream endpoint behaves.
How can we reproduce it (as minimally and precisely as possible)?
Add decisions with
-
cscli decision add -i 1.2.3.4 -
cscli decision add -r 1.2.3.0/24 -
cscli decision add --scope UserAgent --value crowdsec
Register an API key
-
cscli bouncer add MyTestBouncer
Use that API key to cURL the getDecisionsStream endpoint:
-
curl -sS -H "X-Api-Key: 9E1uMkxZzSDYQZU/Qbm9Og" -k 'https://127.0.0.1:8080/v1/decisions?origins=cscli%2Ccrowdsec' | jq .
Notice that the custom scope (UserAgent) decision is missing from the list returned by getDecisionsStream
Anything else we need to know?
No response
Crowdsec version
$ cscli version
2023/11/27 13:00:52 version: v1.5.5-d2d788c5dc0a9e387635276623c6781774a9dfd4
2023/11/27 13:00:52 Codename: alphaga
2023/11/27 13:00:52 BuildDate: 2023-10-24_08:13:35
2023/11/27 13:00:52 GoVersion: 1.21.3
2023/11/27 13:00:52 Platform: docker
2023/11/27 13:00:52 libre2: C++
2023/11/27 13:00:52 Constraint_parser: >= 1.0, <= 2.0
2023/11/27 13:00:52 Constraint_scenario: >= 1.0, < 3.0
2023/11/27 13:00:52 Constraint_api: v1
2023/11/27 13:00:52 Constraint_acquis: >= 1.0, < 2.0
OS version
# On Linux:
$ cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.18.4
PRETTY_NAME="Alpine Linux v3.18"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
$ uname -a
Linux ip-172-18-18-8 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 Linux
Enabled collections and parsers
$ cscli hub list -o raw
# paste output here
Acquisition config
Config show
$ cscli config show
Global:
- Configuration Folder : /etc/crowdsec
- Data Folder : /var/lib/crowdsec/data
- Hub Folder : /etc/crowdsec/hub
- Simulation File : /etc/crowdsec/simulation.yaml
- Log Folder : /var/log/
- Log level : info
- Log Media : file
Crowdsec:
- Acquisition File : /etc/crowdsec/acquis.yaml
- Parsers routines : 1
- Acquisition Folder : /etc/crowdsec/acquis.d
cscli:
- Output : human
- Hub Branch :
- Hub Folder : /etc/crowdsec/hub
API Client:
- URL : https://127.0.0.1:8080/
- Login :
- Credentials File : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
- Listen URL : 0.0.0.0:8080
- Profile File : /etc/crowdsec/profiles.yaml
- Cert File : /tmp/server.pem
- Key File : /tmp/server-key.pem
- CA Cert : /tmp/ca_bundle.pem
- Allowed Agents OU : Freelancer Ltd
- Allowed Bouncers OU : Freelancer Ltd
- Trusted IPs:
- 127.0.0.1
- ::1
- Database:
- Type : mysql
- Host : db-crowdsec.syd1.fln-dev.flnltd.com
- Port : 3306
- User : crowdsec
- DB Name : crowdsec
- Flush age : 7d
- Flush size : 5000
Prometheus metrics
$ cscli metrics
Local API Metrics:
╭──────────────────────┬────────┬──────╮
│ Route │ Method │ Hits │
├──────────────────────┼────────┼──────┤
│ /v1/alerts │ GET │ 4 │
│ /v1/alerts │ POST │ 5 │
│ /v1/decisions │ GET │ 10 │
│ /v1/decisions/stream │ GET │ 244 │
│ /v1/heartbeat │ GET │ 78 │
│ /v1/watchers/login │ POST │ 12 │
╰──────────────────────┴────────┴──────╯
Local API Machines Metrics:
╭──────────────────────────────┬───────────────┬────────┬──────╮
│ Machine │ Route │ Method │ Hits │
├──────────────────────────────┼───────────────┼────────┼──────┤
│ [email protected] │ /v1/alerts │ GET │ 2 │
│ [email protected] │ /v1/alerts │ POST │ 4 │
│ [email protected] │ /v1/heartbeat │ GET │ 37 │
│ [email protected] │ /v1/alerts │ GET │ 2 │
│ [email protected] │ /v1/alerts │ POST │ 1 │
│ [email protected] │ /v1/heartbeat │ GET │ 37 │
╰──────────────────────────────┴───────────────┴────────┴──────╯
Local API Bouncers Metrics:
╭────────────────────────────┬──────────────────────┬────────┬──────╮
│ Bouncer │ Route │ Method │ Hits │
├────────────────────────────┼──────────────────────┼────────┼──────┤
│ MyTestClient │ /v1/decisions/stream │ GET │ 12 │
│ MyTestClient │ /v1/decisions │ GET │ 10 │
│ [email protected] │ /v1/decisions/stream │ GET │ 232 │
╰────────────────────────────┴──────────────────────┴────────┴──────╯
Local API Bouncers Decisions:
╭──────────────┬───────────────┬───────────────────╮
│ Bouncer │ Empty answers │ Non-empty answers │
├──────────────┼───────────────┼───────────────────┤
│ MyTestClient │ 0 │ 10 │
╰──────────────┴───────────────┴───────────────────╯
Local API Decisions:
╭────────────────────────────────────────────┬──────────┬─────────┬───────╮
│ Reason │ Origin │ Action │ Count │
├────────────────────────────────────────────┼──────────┼─────────┼───────┤
│ crowdsecurity/f5-big-ip-cve-2020-5902 │ CAPI │ ban │ 22 │
│ crowdsecurity/ssh-slow-bf │ CAPI │ ban │ 12 │
│ crowdsecurity/CVE-2022-41082 │ CAPI │ ban │ 367 │
│ crowdsecurity/CVE-2022-42889 │ CAPI │ ban │ 4 │
│ crowdsecurity/http-cve-2021-41773 │ CAPI │ ban │ 20 │
│ crowdsecurity/http-path-traversal-probing │ CAPI │ ban │ 24 │
│ crowdsecurity/CVE-2022-37042 │ CAPI │ ban │ 14 │
│ crowdsecurity/CVE-2023-22518 │ CAPI │ ban │ 15 │
│ crowdsecurity/http-backdoors-attempts │ CAPI │ ban │ 255 │
│ crowdsecurity/netgear_rce │ CAPI │ ban │ 6 │
│ flnltd/http-bruteforce-login │ crowdsec │ captcha │ 1 │
│ crowdsecurity/apache_log4j2_cve-2021-44228 │ CAPI │ ban │ 306 │
│ crowdsecurity/http-probing │ CAPI │ ban │ 742 │
│ crowdsecurity/grafana-cve-2021-43798 │ CAPI │ ban │ 57 │
│ crowdsecurity/http-sensitive-files │ CAPI │ ban │ 8 │
│ crowdsecurity/jira_cve-2021-26086 │ CAPI │ ban │ 19 │
│ manual 'ban' from '' │ cscli │ ban │ 4 │
│ crowdsecurity/thinkphp-cve-2018-20062 │ CAPI │ ban │ 14 │
│ crowdsecurity/CVE-2019-18935 │ CAPI │ ban │ 32 │
│ crowdsecurity/CVE-2022-35914 │ CAPI │ ban │ 45 │
│ crowdsecurity/fortinet-cve-2018-13379 │ CAPI │ ban │ 76 │
│ crowdsecurity/http-bad-user-agent │ CAPI │ ban │ 2402 │
│ crowdsecurity/http-crawl-non_statics │ CAPI │ ban │ 147 │
│ crowdsecurity/http-open-proxy │ CAPI │ ban │ 267 │
│ crowdsecurity/nginx-req-limit-exceeded │ CAPI │ ban │ 47 │
│ crowdsecurity/CVE-2022-26134 │ CAPI │ ban │ 165 │
│ crowdsecurity/CVE-2023-22515 │ CAPI │ ban │ 4 │
│ crowdsecurity/http-generic-bf │ CAPI │ ban │ 8 │
│ crowdsecurity/ssh-bf │ CAPI │ ban │ 9922 │
╰────────────────────────────────────────────┴──────────┴─────────┴───────╯
Local API Alerts:
╭──────────────────────────────┬───────╮
│ Reason │ Count │
├──────────────────────────────┼───────┤
│ flnltd/http-bruteforce-login │ 2 │
│ manual 'ban' from '' │ 4 │
╰──────────────────────────────┴───────╯
Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.
@laurentgoudet: Thanks for opening an issue, it is currently awaiting triage.
In the meantime, you can:
- Check Crowdsec Documentation to see if your issue can be self resolved.
- You can also join our Discord.
- Check Releases to make sure your agent is on the latest version.
Details
I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hi thank you for your report,
We intentionally set those scopes if non is provided
https://github.com/crowdsecurity/crowdsec/blob/67cdf91f94a16fdb7122c2e450addd88c931ca6c/pkg/apiserver/controllers/v1/decisions.go#L353-L355
I will ask the team if this should be the same as non stream or if we should not do this.
Thanks. Arguably the inconsistency between getDecisions & getDecisionsStream is just a minor annoyance, what I am really trying to achieve here is being able to use https://github.com/crowdsecurity/cs-custom-bouncer with custom scopes, without having to rebuild the whole thing myself just to pass a custom scopes filter.
Thanks. Arguably the inconsistency between
getDecisions&getDecisionsStreamis just a minor annoyance, what I am really trying to achieve here is being able to use https://github.com/crowdsecurity/cs-custom-bouncer with custom scopes, without having to rebuild the whole thing myself just to pass a customscopesfilter.
You shouldnt have to rebuild it, our go SDK already has scopes yaml keys.
https://github.com/crowdsecurity/go-cs-bouncer/blob/ff0ac9af53a6110b88e415b0a0e876cfc82341ea/stream_bouncer.go#L39C2-L39C8
So within all bouncers you can add the following to the configuration as an example
scopes:
- ip
- range
- useragent