beats icon indicating copy to clipboard operation
beats copied to clipboard

7.17 filebeat `elasticsearch.slowlog` ingested log entry has full json log as message

Open gigerdo opened this issue 8 months ago • 0 comments

When using filebeat to ingest ES slowlogs, the resulting document does not have the correct message. Instead the message contains the full JSON log entry:

image

Reproduction:

  • Version 7.17.21 for Elasticsearch and Filebeat
  • Setup filebeat to ingest slowlogs using the elasticsearch.slowlog module (For example by using the log+metrics feature in elastic cloud)

Here an example of a log entry as it appears in the ES log file:

{"type": "index_search_slowlog", "timestamp": "2024-06-13T11:54:45,125Z", "level": "WARN", "component": "i.s.s.query", "cluster.name": "13fd6fa94ab840c088e86a8cd8faa3b8", "node.name": "instance-0000000000", "message": "[kibana_sample_data_ecommerce][0]", "took": "5.5ms", "took_millis": "5", "total_hits": "0+ hits", "types": "[]", "stats": "[]", "search_type": "QUERY_THEN_FETCH", "total_shards": "1", "source": "{\"size\":500,\"query\":{\"bool\":{\"filter\":[{\"range\":{\"order_date\":{\"from\":\"2024-06-13T11:39:45.023Z\",\"to\":\"2024-06-13T11:54:45.023Z\",\"include_lower\":true,\"include_upper\":true,\"format\":\"strict_date_optional_time\",\"boost\":1.0}}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"version\":true,\"_source\":false,\"stored_fields\":\"*\",\"fields\":[{\"field\":\"*\",\"include_unmapped\":true},{\"field\":\"customer_birth_date\",\"format\":\"strict_date_optional_time\"},{\"field\":\"order_date\",\"format\":\"strict_date_optional_time\"},{\"field\":\"products.created_on\",\"format\":\"strict_date_optional_time\"}],\"script_fields\":{},\"sort\":[{\"order_date\":{\"order\":\"desc\",\"unmapped_type\":\"boolean\"}}],\"track_total_hits\":-1,\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fragment_size\":2147483647,\"fields\":{\"*\":{}}}}", "id": "b060ba95-ddea-4aa2-8a36-82a7a36b1564", "cluster.uuid": "UgI0rr5lSqeQtW98mENVxQ", "node.id": "BbJ5ZoU7QE2Q-U6fMAXgng" , "trace.id": "e738203f84d7190e649598ee3a54152d"  }

gigerdo avatar Jun 18 '24 13:06 gigerdo