Allow VictoriaLogs data source to dynamically set the type label (and maybe add some parsed data)

[thebondo]

/kind enhancement

What would you like to be added?

Add a mechanism for setting event.Line.Labels.type dynmically based on the query response from VictoraLogs for each event.

Add a mechanism to include other results from the VictoriaLogs JSON in the event.Parsed map.

Why is this needed?

The s00-raw parser crowdsecurity/non-syslog that is included with the crowdsecurity/linux collection takes the event.Line.Labels.type value and puts it into event.Parsed.program. This value is then used for selected the parser for the next stage. Currently, the VictoriaLogs data source uses a static Labels map from the acuisition configuration to set a single static value for the type.

If you have a VictoriaLogs instance that collects logs from multiple types of sources, then you currently would need to configure multiple CrowdSecurity data sources, using a separate query and Labels.type value to ingest them into CrowdSecurity. This could be simplified by allowing the data source to dynamically set the Labels.type value for each event dynamically based on a value from the full JSON value returned by the VictoriaLogs query.

So instead of something like in /etc/crowdsec/acuis.yaml

source: victorialogs
mode: tail
url: http://127.0.0.1:9428
query: 'service:sshd'
labels:
  type: sshd
---
source: victorialogs
mode: tail
url: http://127.0.0.1:9428
query: 'service:nginx'
labels:
  type: nginx

requiring a separate configuration for every service, you could have something like

source: victorialogs
mode: tail
url: http://127.0.0.1:9428
query: '*'
label_key: service

and then have the data source code automatically set event.Line.Labels.type to the value of service from the JSON blob that was received for event.

It might also be helpful to make more of the parsed JSON received from VictoriaLogs available in CrowdSecurity by copying keys into event.Parsed.

Note: I have already created working code that maps JSON results from VictoriaLogs into event.Parsed with a configurable remapping of names and works together with a new s00-raw parser for getting the event to the appropriate parser at the next stage. But my approach might be a bit more general than is desired, and at least being able to set the type dynamically would be awesome.

[github-actions[bot]]

@thebondo: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions[bot]]

@thebondo: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[thebondo]

/kind enhancement