beats
beats copied to clipboard
7.17 filebeat `elasticsearch.slowlog` ingested log entry has full json log as message
When using filebeat to ingest ES slowlogs, the resulting document does not have the correct message. Instead the message contains the full JSON log entry:
Reproduction:
- Version
7.17.21for Elasticsearch and Filebeat - Setup filebeat to ingest slowlogs using the
elasticsearch.slowlogmodule (For example by using the log+metrics feature in elastic cloud)
Here an example of a log entry as it appears in the ES log file:
{"type": "index_search_slowlog", "timestamp": "2024-06-13T11:54:45,125Z", "level": "WARN", "component": "i.s.s.query", "cluster.name": "13fd6fa94ab840c088e86a8cd8faa3b8", "node.name": "instance-0000000000", "message": "[kibana_sample_data_ecommerce][0]", "took": "5.5ms", "took_millis": "5", "total_hits": "0+ hits", "types": "[]", "stats": "[]", "search_type": "QUERY_THEN_FETCH", "total_shards": "1", "source": "{\"size\":500,\"query\":{\"bool\":{\"filter\":[{\"range\":{\"order_date\":{\"from\":\"2024-06-13T11:39:45.023Z\",\"to\":\"2024-06-13T11:54:45.023Z\",\"include_lower\":true,\"include_upper\":true,\"format\":\"strict_date_optional_time\",\"boost\":1.0}}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"version\":true,\"_source\":false,\"stored_fields\":\"*\",\"fields\":[{\"field\":\"*\",\"include_unmapped\":true},{\"field\":\"customer_birth_date\",\"format\":\"strict_date_optional_time\"},{\"field\":\"order_date\",\"format\":\"strict_date_optional_time\"},{\"field\":\"products.created_on\",\"format\":\"strict_date_optional_time\"}],\"script_fields\":{},\"sort\":[{\"order_date\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"track_total_hits\":-1,\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fragment_size\":2147483647,\"fields\":{\"*\":{}}}}", "id": "b060ba95-ddea-4aa2-8a36-82a7a36b1564", "cluster.uuid": "UgI0rr5lSqeQtW98mENVxQ", "node.id": "BbJ5ZoU7QE2Q-U6fMAXgng" , "trace.id": "e738203f84d7190e649598ee3a54152d" }
Hi! We just realized that we haven't looked into this issue in a while. We're sorry!
We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!