Provide better Grafana Support

[sts]

As a WAF admin I would like to gain insights into Coraza activity, which requests were blocked, and for what reason. I'd like to be able to use widespread Grafana for that purpose and store my logs in Elasticsearch or Loki.

Coraza currently provides a JSON log format, which is unsuitable for both backends for the following reasons:

• Grafana/Elasticsearch: Grafana is unable to perform aggregations on nested fields with an Elasticsarch backend. Issue #4693 -> Nested objects should be reduced by using field name prefixing. So instead of using:

{
 "transaction": {
   "timestamp": "02/Jan/2006:15:04:20 -0700",
   "id": "ABCDEFGHIJKMNLAB",
 }
}

Use this:

{ "transaction_timestamp": "02/Jan/2006:15:04:20 -0700",
  "transaction_id": "ABCDEFGHIJKMNLAB",
}

• Grafana/Loki: Grafana is unable to perform searches with fields stored in json arrays. -> messages[] should be dissolved in single log entries, which can be queried by looking up a transaction_id and a log type. So instead of using:

{ "messages": [{
    "actionset": "Warning",
    "message": "Pattern match \"\\\\< ?script\\\\b\" at ARGS_NAMES:<script.",
    "data": {
	    "file": "/etc/coraza/crs/rules.conf"
}

use this:

{ "type": "message",
  "transaction_id": "ABCDEFGHIJKMNLAB",
  "actionset": "Warning",
  "message": "Pattern match \"\\\\< ?script\\\\b\" at ARGS_NAMES:<script.",
  "data_file": "/etc/coraza/crs/rules.conf"
}

There is a initial logging POC which verifies logging messages apart from the transaction works with Grafana https://github.com/sts/coraza-logging

[jptosso]

Looks good to me. Could you describe a struct of the expected fields? I think we won't have a problem merging it and saving it as SecAuditLogFormat KEY_VALUE_JSON or something like that. Still, we usually expect users to use something like logstash to rewrite the logs.

[jptosso]

V3 includes a map flattening utility that can be used to easily achieve this, it would be as simple as: encoding.MapFlatten(auditLog)

[sts]

I still need to do some testing on the optimal format. I also launched a test with Elaticsearch as a backend, as Grafana has better support for Elastic. So this will still need some time.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[jptosso]

Waiting approval on https://github.com/corazawaf/coraza/pull/294

[jcchavezs]

I'd love to hear from @piyushroshan and @fzipi on this.

[ethan-gallant]

I'm going to be pushing all month to get this merged which would remove the need for this change: https://github.com/grafana/grafana/pull/47233

Any additional support on the issue so Gafana doesn't shrug it off like they did two years ago is greatly appreciated.

[jptosso]

Thank you for the update @ethan-gallant , that will be an excellent grafana feature.

In the meantime, we could split this into a plugin and close the active PR. What do you think, @sts?

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.