beats icon indicating copy to clipboard operation
beats copied to clipboard

7.17 filebeat `elasticsearch.slowlog` ingested log entry has full json log as message

Open gigerdo opened this issue 1 year ago • 1 comments

When using filebeat to ingest ES slowlogs, the resulting document does not have the correct message. Instead the message contains the full JSON log entry:

image

Reproduction:

  • Version 7.17.21 for Elasticsearch and Filebeat
  • Setup filebeat to ingest slowlogs using the elasticsearch.slowlog module (For example by using the log+metrics feature in elastic cloud)

Here an example of a log entry as it appears in the ES log file:

{"type": "index_search_slowlog", "timestamp": "2024-06-13T11:54:45,125Z", "level": "WARN", "component": "i.s.s.query", "cluster.name": "13fd6fa94ab840c088e86a8cd8faa3b8", "node.name": "instance-0000000000", "message": "[kibana_sample_data_ecommerce][0]", "took": "5.5ms", "took_millis": "5", "total_hits": "0+ hits", "types": "[]", "stats": "[]", "search_type": "QUERY_THEN_FETCH", "total_shards": "1", "source": "{\"size\":500,\"query\":{\"bool\":{\"filter\":[{\"range\":{\"order_date\":{\"from\":\"2024-06-13T11:39:45.023Z\",\"to\":\"2024-06-13T11:54:45.023Z\",\"include_lower\":true,\"include_upper\":true,\"format\":\"strict_date_optional_time\",\"boost\":1.0}}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"version\":true,\"_source\":false,\"stored_fields\":\"*\",\"fields\":[{\"field\":\"*\",\"include_unmapped\":true},{\"field\":\"customer_birth_date\",\"format\":\"strict_date_optional_time\"},{\"field\":\"order_date\",\"format\":\"strict_date_optional_time\"},{\"field\":\"products.created_on\",\"format\":\"strict_date_optional_time\"}],\"script_fields\":{},\"sort\":[{\"order_date\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"track_total_hits\":-1,\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fragment_size\":2147483647,\"fields\":{\"*\":{}}}}", "id": "b060ba95-ddea-4aa2-8a36-82a7a36b1564", "cluster.uuid": "UgI0rr5lSqeQtW98mENVxQ", "node.id": "BbJ5ZoU7QE2Q-U6fMAXgng" , "trace.id": "e738203f84d7190e649598ee3a54152d"  }

gigerdo avatar Jun 18 '24 13:06 gigerdo

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic[bot] avatar Jun 18 '25 13:06 botelastic[bot]