AbuseIP enrichment

[maxdd]

What would you like to be added?

/kind feature I've recently opened a post here

https://discourse.crowdsec.net/t/abuseip-confidence-score-and-total-reports-enricher/2731

but the forum seems less populated so i will try here.

The feature i would like to see is an enrichment that either generically allow a script to be executed or a dedicated one for abuseip that it adds TotalReports and Score to the metadata so that it can be sent as part of a notification.

I'm willing to experiment but im struggling to understand or find examples on how to implement an enrich feature. Any guideline you can provide?

Why is this needed?

I'm monitoring cyber threats activities and i find it really useful to have an IP score.

[github-actions[bot]]

@maxdd: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions[bot]]

@maxdd: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[LaurenceJJones]

Hey thank you for opening a feature request, however, we see very little value in making CrowdSec become a "must be able to do every enrichment tool". Cause it increases code debt for maybe a handful of users that actually use the functionality, if the providers ever change format or how we can get the data then it breaks then users of the function will then say "you must release asap cause this functionality is broken".

also allowing a process to execute a sh script is a security nightmare (user leaves rw permissions, somebody places a malicious line and bang we have code execution from a process owner)

So you can achieve this function by either setting your own http server inbetween to do the additional functionality EG: crowdsec -> express app -> prometheus or if prometheus (like other SIEMs like wazuh) offers enrichment support on their side then you can simply setup a playbook (SIEM terminology) after receiving the request to enrich the event with your own data.

You could also write your own plugin as well but the plugin only gets access to the string message and not to the actual alert model so it would mean you would have to parse the string alert to an alert and will become very messy not to mention it not that clear how to implement your own plugin at the moment so I would say the above options are much better.

So for now will be closing as not planned for the above reasons.

[maxdd]

Hello Laurence, thank you for the answer. I was under the impression that this is no different than geoip-enrich module in terms of philosophy. I agree that having it as part of a standard deployment doesnt make too much sense and i was exactly hoping for a plugin integration. Having to host a REST API with an underline script execution is not that bad but it feels like it wont seamlessly integrate with the idea of the crowdsec hub you have. What are you referring to with

not to mention it not that clear how to implement your own plugin at the moment ?

My idea was to simply add two fields to the alert model in the same way it is done for geoip-enrich i guess

In order to use an express app i would need an http notification right? Since i'm already configuring something similar for the prometheus notification i think the idea would be to change the ip to my express app and have that one relay the info to prometheus, am i correct?

[LaurenceJJones]

What are you referring to with

not to mention it not that clear how to implement your own plugin at the moment ?

We have a plugin system https://docs.crowdsec.net/docs/next/local_api/notification_plugins/intro that you can write your own plugin for example like user https://github.com/zbalkan/notification-file did before there was ample demand for it to be introduced into our repository.

but the main problem is the plugin system only sends the message key which is a string. So for you to be able to enrich the data you would have to convert the string back to an alert.

So the easiest solution is pointing the http plugin to an external http service like an express app which accepts the data, reads the JSON and then adds the enriched data you would like to then forward to prometheus.

[maxdd]

Im struggling to understand the string / alert thing. My notification for prometheus is as follow

type: http
name: http_victoriametrics
log_level: debug
format: >
  {{- range $Alert := . -}}
  {{- $traefikRouters := GetMeta . "traefik_router_name" -}}
  {{- range .Decisions -}}
  {"metric":{"__name__":"cs_lapi_decision","instance":"server","country":"{{$Alert.Source.Cn}}","asname":"{{$Alert.Source.AsName}}","asnumber":"{{$Alert.Source.AsNumber}}","latitude":"{{$Alert.Source.Latitude}}","longitude":"{{$Alert.Source.Longitude}}","iprange":"{{$Alert.Source.Range}}","scenario":"{{.Scenario}}","type":"{{.Type}}","duration":"{{.Duration}}","scope":"{{.Scope}}","ip":"{{.Value}}","traefik_routers":{{ printf "%q" ($traefikRouters | uniq | join ",")}}},"values": [1],"timestamps":[{{now|unixEpoch}}000]}
  {{- end }}
  {{- end -}}
url: http://victoriametrics:8428/api/v1/import
method: POST
headers:
  Content-Type: application/json
  # if you use vmauth as proxy, please uncomment next line and add your token
  #Authorization: "<SECRET-TOKEN>"

You are saying the plugin is not receiving the $Alert object but basically a formatted json string? Why can i parse it back to a json object?

[LaurenceJJones]

So in our plugin system the format key is a template that is ran inside CrowdSec and plugin recieves the formatted string EG: if the template is test it will receive "test", know the example is not an actual use case but it to show that it gets passed as a string. That means in our plugins we cannot parse it from a string back to an alert because its a string, you could do json -> alert but since Go is a strict type system you have to keep the struct references in parallel to the crowdsec version.

Now in javascript they dont care about types so if you have an express app, it can just parse the json data with no issue cause it doesnt care about the types. The you can extract the IP, do your enriching and then put it all back together to pass to prometheus.