Blank rule after applying manual filter to new term alert

[scott-oit]

If I create a new rule with the New Term event and apply a manual filter, I am not able to edit the rule after saving. If I select the rule in Praeco, I get a blank screen. The YAML looks fine. I used a simple rule in the "field : value" format.

As a work around, use the Builder. However, you are not able to use OR statements.

[daichi703n]

Editing the rule directly is deprecated.

For OR rule, doesn't it work like this?

Generated YAML:

__praeco_full_path: monitoring-elk/test.new_term
__praeco_query_builder: >-
  {"query":{"logicalOperator":"any","children":[{"type":"query-builder-rule","query":{"rule":"host","selectedOperator":"contains","selectedOperand":"host","value":"test"}},{"type":"query-builder-rule","query":{"rule":"error_message","selectedOperator":"contains","selectedOperand":"error_message","value":"fatal"}}]}}
alert:
  - slack
alert_on_missing_field: false
alert_subject: New Term Found
alert_subject_args: []
filter:
  - query_string:
      query: 'host:test OR error_message:fatal'
import: ../BaseRule.config
...

[scott-oit]

Using the manual filter breaks the UI for me. Using the Builder only works just fine. By editing the rule, do you mean by editing YAML file? If so, I didn't do that. Only viewed it. Here is my yaml if that's helpful:

aggregation:
  schedule: 10 * * * *
aggregation_key: server.ip
alert:
  - email
alert_on_missing_field: false
alert_subject: "Port {0}\_open on {1}\_({2})"
alert_subject_args:
  - server.port
  - server.domain
  - server.ip
alert_text_type: alert_text_only
email:
  - [email protected]
filter:
  - query_string:
      query: >-
        nmap.port.state: open AND network.name : Management OR network.name :
        Server OR network.name : Printer OR network.name : Voice
from_addr: [email protected]
import: ../../BaseRule.config
index: nmap-*
is_enabled: true
name: NMAP New Port
query_key: nmap.fingerpint.keyword
realert:
  minutes: 0
summary_table_fields:
  - server.ip
  - server.domain
  - server.port
terms_window_size:
  days: 30
timestamp_field: '@timestamp'
timestamp_type: iso
type: new_term
use_keyword_postfix: true
use_strftime_index: false
window_step_size:
  days: 1

[daichi703n]

I see. I reproduced your issue.

~~I think this is a bug on generating new_term rule. I'm continue to specify where is the bug happen. (Praeco? ElastAlert Server?)~~

~~By my testing, generated rule.yml has invalid syntax about filter.~~

ElastAlert rule for Elasticsearch is this.

$ cat test.new_term.yaml
...
filter:
  - query_string:
      query: 'host:test'
import: ../BaseRule.config
...

But, Praeco GUI expects this format.

...
filter:
  - query:
      query_string:
        query: 'host:test'
import: ../BaseRule.config
...

(debugging only, not workaround) ~~Modifying rule.yaml on Server fixes the displaying on GUI (this fix isn’t applied from GUI). Therefore, the root cause of this issue is seems to be correct.~~ This patch enables to render GUI, but ElastAlert cannot run query (invalid syntax).

(workaround) By using Builder,

__praeco_query_builder: '{"query":{"logicalOperator":"all","children":[]}}'

is added into rule.yml, and skip the filter: section on loading, this will not break the GUI.

[daichi703n]

@johnsusek Do you already know this problem?

[daichi703n]

@scott-oit I found that generated rule.yaml is correct for ElastAlert, but Praeco cannot render this syntax pattern.

filter syntax for new_term is discussed in here. https://github.com/Yelp/elastalert/issues/2689#issuecomment-590886923

[scott-oit]

Thanks for looking into this. Are you saying that the generated file is correct, but Praeco can't present it in the UI?

[daichi703n]

Yes. But, I don’t think this is a bug, because keeping common handling each rule helps to make it easy to maintain (prevent future bug) the Praeco. It seems to be better that ElastAlert to fix the new_term query schema same as other. At a moment, I don’t have an idea how to fix this...

[johnsusek]

I will look into the issue with manual filters, they should work

[nsano-rururu]

@daichi703n

以下のissueも同じような問題でしょうか？

https://github.com/Yelp/elastalert/issues/1045#issuecomment-296257584

This was fixed in master yesterday: 95b3f0a

To work around, change your filter to

filter:
 - query_string:
      query: "_exists_:$event.supplier"

EDIT: This is a different issue. You're not supposed to have a - before query_string when it's not the top level. The fix is still the filter I posted.