alerting-dashboards-plugin
alerting-dashboards-plugin copied to clipboard
opensearch-project
[BUG] Visual editor Group By on per bucket monitor not showing fields
What is the bug? Group by not working on per bucket monitor visual editor
How can one reproduce the bug? Steps to reproduce the behavior:
- Go to 'Alerting'
- Click on 'Create Monitor'
- Select Per Bucket Monitor
- Select visual Editor
- Select index
- Scroll down to "Query"
- Try to select a group, you will see "You've selected all available options", no suggestions on fields, no way to select the field manually
What is the expected behavior? You can see suggestion fields or manually type a field name
What is your host/environment?
- OS 2.0
Do you have any screenshots?
Hi @llermaly, would it be possible for you to provide the mapping of the index you're using for this example? Feel free to obfuscate any names if you like, the mapping field types is what I'm looking for.
Hello @qreshi , I'm using the built in security index for audit logs.
security-auditlog-*
@qreshi Mappings attached. I didnt change anything, just want to create an alert using the visual editor to alert on more than X login failed attempts grouped by user:
{
"security-auditlog-2022.06.20" : {
"mappings" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"audit_category" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_cluster_name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_format_version" : {
"type" : "long"
},
"audit_node_host_address" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_node_host_name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_node_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_node_name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_request_body" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_request_effective_user" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_request_effective_user_is_admin" : {
"type" : "boolean"
},
"audit_request_layer" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_request_origin" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_request_privilege" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_request_remote_address" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_rest_request_headers" : {
"properties" : {
"Connection" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"Host" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"content-length" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"x-opaque-id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"x-opensearch-product-origin" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"audit_rest_request_method" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_rest_request_path" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_trace_doc_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_trace_indices" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_trace_resolved_indices" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_trace_task_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_trace_task_parent_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"audit_transport_headers" : {
"properties" : {
"X-Opaque-Id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"_opendistro_security_initial_action_class_header" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"_opendistro_security_origin_header" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"_opendistro_security_remote_address_header" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"_opendistro_security_remotecn" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"_opendistro_security_user_header" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"_system_index_access_allowed" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"audit_transport_request_type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
For me, this also happens in the "Per Query Monitor"
For me it was an error in my mapping. I had an index name without the filebeat version so the mapping template was not used.
Before: filebeat-2020.08
After: filebeat-7.12.1-2020.08
So this is solved for me.
Tried to reproduce it, but it seems to be working for both monitor types
Per query monitor:
Per bucket monitor:
Same here. Is a template an absolute requirement for "Group by"? It would be great if one could use it for index patterns as well.
In the GroupBy UI control we allow only fields that have type "keyword". @llermaly In the above mapping I don't see a field with keyword type, hence the empty control.
@qreshi should this be made more apparent in the UX? or should we disable the control when there are no keyword type fields?
@amsiglan Yeah, I think we should make the current requirement more apparent in the UX.
Also the keyword filter is because the underlying aggregation is a term aggregation. I believe term aggregations support keyword, numeric, ip, boolean, or binary types. So we should test those other types out in the backend to ensure they work as expected and then update the frontend to allow all of those types as well.
Looked a little more into the mapping shared above and it does have the subfields to specify the keyword type for certain fields which I guess should be considered when extracting the fields as part of getTypeFromMappings.
So in addition to exposing more types, will also check if extracting the subfield type works.
I think I have the same problem. I'm using metricbeat and I'm trying to group a metric by kubernetes.node.name (or kubernetes.node.name.keyword) and it won't let me. According to GET logstash-metrics-*/_mapping/, it should work:
{
"logstash-metrics-2022.12.21" : {
"mappings" : {
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"norms" : false,
"type" : "text"
}
}
},
{
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
},
"norms" : false,
"type" : "text"
}
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "keyword"
},
"kubernetes" : {
"properties" : {
"node" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
},
"norms" : false
}
}
}
}
}
}
}
}
}
Hello, I'm facing the same problem, any update on the resolution of the bug? Kind regards,
I was able to work around the issue by constructing a POST request manually.
I'm facing the same issue even if my mapping has a couple of keyword fields !!!
any fix or workaround pls
I was able to work around the issue by constructing a POST request manually.
Hi @BalzGuenat i have the same issue as yours was unable to set group by through UI. can u please past an example of POST request u did. i am trying to find one online which uses a group by but unable to find one. Thanks!
The following enhancement issue tracks the request to support more types than just keyword in the Group by dropdown. It also has an example of a workaround that involves using the extraction query editor UI.
https://github.com/opensearch-project/alerting-dashboards-plugin/issues/230
An example of a POST command can be found in the OpenSearch documentation as well for those interested.
https://opensearch.org/docs/latest/observing-your-data/alerting/api/#create-a-bucket-level-monitor
This is still an issue, quite a glaring standing omission that used to work just fine.
We're having to define a lot of our queries manually due to this bug, which is quite cumbersome.
This is still an issue, quite a glaring standing omission that used to work just fine.
We're having to define a lot of our queries manually due to this bug, which is quite cumbersome.
2nded
The team has been busy with other commitments, we will pick this up in a week or so, but please feel free to submit a PR with the fix. Thank you for the patience.
Hello @amsiglan,
is there any update on the fix? if not then can we remove this group by filter mandatory check in version - 2.16.0?
