Elastalert group by client_id

[F1sn1k]

Hello. Is there a way I can get separate results for different clients with one query

query: 'event: "alert_sent"`

and I like to group by client_id

The desired result I want for example (If Client has less than 10 hits = threshold):

Client A: 8 hits
Client B: 4 hits
Client C: 100 (should not alert)

Using query_key: client_id I get a result with list of clients but all have same result for example:

Client A: 100 hits
Client B: 100 hits
Client C: 100 hits

---

Here is sample of my rule:

index: "mya_output_eb-*"
name: Total alert for clients
timestamp_field: timestamp

alert_text: "There were {0} notifications for {1}."
alert_text_args: ["num_hits", "client_id"]
alert_text_type: alert_text_only

type: flatline
threshold: 10
timeframe:
  hours: 24
 
filter:
- query:
    query_string:
      query: 'event: "alert_sent"'

query_key: client_id

alert:
  - slack
slack_webhook_url: https://hooks.slack.com/services/xxx/xxx/xxx

Should I use aggregation with aggregation keys? or other configuration ?

[F1sn1k]

@Qmando can you help pls

[Qmando]

Your rule looks fine. You say "all have same result", how do you know that? Elastalert doesn't tell you the numbers for each value. It's probably working correctly. Note that you must leave it running for at least 24 hours (timeframe) before it will alert.

If you want to test it without waiting that long, try adding a start parameter of more than 1 day ago, ie --start 2018-02-21. If are more than 1 but less than 10 of any client_id from the last 24 hours, it should alert on them.

[F1sn1k]

In Elasticsearch I get desired results:

GET /logstash-*/_search
{
  "size": 0,
  "aggregations": {
    "total_per_client": {
      "terms": {
        "field": "client_id"
      }
    }
  },
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "alert_sent",
            "fields": [
              "event"
            ]
          }
        }
      ],
    }
  }
}

AND the result is what I want:

  "aggregations": {
    "total_per_client": {
      "doc_count_error_upper_bound": 6,
      "sum_other_doc_count": 42998,
      "buckets": [
        {
          "key": "client-A",
          "doc_count": 4315971
        },
        {
          "key": "client-B",
          "doc_count": 1828976
        },
        {
          "key": "client-C",
          "doc_count": 10
        },
         ...
      ]
    }
  }

I cant get this results with ElastAlert.

Lastly I tried:

use_terms_query: true query_key: client_id doc_type: logs ( maybe this is wrong. How can I see the doc_type? )

AND now I use frequency since use_terms_query is not used with flatline according to documentation

[Qmando]

You CAN use use_terms_query with flatline type. It's in http://elastalert.readthedocs.io/en/latest/ruletypes.html#flatline, where did you see it couldn't be used together?

"I cant get this results with ElastAlert." You can get exactly those results.

doc_type is the _type field of a document.

[F1sn1k]

--es_debug_trace logtest helped me a lot. doc_type is fixed as well thank you.

The only problem now is how to get the aggregation value.

Here is my title I want to display:

alert_text: "There were {0} notifications for {1}."
alert_text_args: ["num_hits", "client_id"]
alert_text_type: alert_text_only

Instead of num_hits I want doc_type:

In ELASTIC SEARCH I get:

  "aggregations": {
    "total_per_client": {
      "doc_count_error_upper_bound": 6,
      "sum_other_doc_count": 42998,
      "buckets": [
        {
          "key": "client-A",
          "doc_count": 4315971
        },
        {
          "key": "client-B",
          "doc_count": 1828976
        },
        {
          "key": "client-C",
          "doc_count": 10
        },
         ...
      ]
    }
  }

but in ELASTALERT what I can see is

elastalert - {'match_body': {'timestamp': '2018-02-27T19:34:18.363812Z', 'num_matches': 10, 'client_id': u'xxx', 'num_hits': 27}, 'rule_name': 'Total alert for clients', 'alert_time': datetime.datetime(2018, 2, 27, 19, 34, 23, 893064, tzinfo=tzutc()), 'alert_sent': True, 'alert_info': {'slack_username_override': 'elastalert', 'type': 'slack', 'slack_webhook_url': ['https://hooks.slack.com/services/xxx/xxx/xxxx']}}

elastalert_status - {'hits': 27, 'matches': 10, '@timestamp': datetime.datetime(2018, 2, 27, 19, 34, 24, 449126, tzinfo=tzutc()), 'rule_name': 'Total alert for clients', 'starttime': datetime.datetime(2018, 2, 26, 19, 34, 18, 363812, tzinfo=tzutc()), 'endtime': datetime.datetime(2018, 2, 27, 19, 34, 18, 363812, tzinfo=tzutc()), 'time_taken': 5.127768039703369}

And I get in slack the following result:

Total alert for clients
There were 27 notifications for Client A.

Total alert for clients
There were 27 notifications for Client B.

Total alert for clients
There were 27 notifications for Client C.

[smeesheady]

Any Updated on this ?

[sheng-jie]

I met the same problem, anyone could help? Thanks.