elasticsearch
elasticsearch copied to clipboard
elastic
Add min/max range of the event.ingested field to cluster state for searchable snapshots
Add event.ingested min/max range to cluster state for searchable snapshots in order
to support shard skipping on the search coordinator node for event.ingested
like we do for the @timestamp field.
The key motivation is that the Elastic Security solution uses two timestamp fields:
@timestamp which is user populated and event.ingested which is populated by
an ingest processor which tells the time that the event was ingested.
In some cases, queries are filtered by @timestamp, while in some others they are filtered by event.ingested. When data is stored in the cold or frozen tier, we have a shard skipping mechanism on the coordinating node, which allows the coordinating node to skip shards based on min and max value stored in the cluster state for the @timestamp field. This is done to prevent returning irrelevant errors in case shards in frozen/cold are unavailable (as they have no replicas) and queries that don’t target them (as the datetime filter won’t match them) are executed.
This works well for the @timestamp field and the Security solution could benefit from having the same skipping mechanism applied to the event.ingested field.
I am super not keen for transport version to be added to xcontentparser as an int - that has a bad smell about it. Is there somewhere else that could go so its available in the right places?
I am super not keen for transport version to be added to xcontentparser as an int - that has a bad smell about it. Is there somewhere else that could go so its available in the right places?
@thecoop - that commit has been reverted. After discussing with Henning we are not going to add any TransportVersion logic to the XContent config or parsers.