Evaluate the use of ES DataStreams instead of Index

[ricsanfre]

Enhancement Request

Since Elasticsearch 7.9, Data Streams concept was introduced.

A data stream lets you store append-only time series data across multiple indices while giving you a single named resource for requests. Data streams are well-suited for logs, events, metrics, and other continuously generated data. You can use index lifecycle management (ILM) to automate the management of these backing indices. For example, you can use ILM to automatically move older backing indices to less expensive hardware and delete unneeded indices. ILM can help you reduce costs and overhead as your data grows.

Use this new concept to automatically configure retention policies of the data using ILM policies.

References

• Data Streams
• ILM Manage the Index Life Cycle

Implementation Details

Fluend elastic-search-plugin already support DataStreams: https://github.com/uken/fluent-plugin-elasticsearch#configuration---elasticsearch-output-data-stream