Support AWS Elasticsearch UltraWarm Storage

[cameronattard]

What?

Add an option to transition indices in an index set to UltraWarm storage after a specified number of indices. Transitioning indices is as simple as calling POST _ultrawarm/migration/my-index/_warm .

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/ultrawarm.html

Why?

This would allow significantly more affordable long-term storage of logs in Graylog for users of AWS Elasticsearch.

Your Environment

• Graylog Version: 3.0.x
• Elasticsearch Version: 6.8.x

[radykal-com]

As per https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/ultrawarm.html the feature is in preview and has some specific requirements that must be met in order to ultrawarm storage be available:

• You must create a new ES domain with feature enabled
• ES version must be 6.8
• ES domain must be deployed across 3 AZs
• Data nodes can't be t2 family

The option should be available only to ES clusters eligible to use the feature, so some kind of detection mechanism should be developed, maybe the detection will best fit in the aws-plugin.

And maybe wait until the feature is publicly available everywhere

[cameronattard]

Feature is now GA https://aws.amazon.com/about-aws/whats-new/2020/05/aws-announces-amazon-elasticsearch-service-ultrawarm-general-availability/

[alxshr]

Will it be added to any near future releases?

[dlelewski]

Would love this as well - we're moving our entire graylog install to AWS and the cost savings here would be fairly significant

[jpdstan]

+1

[jpdstan]

FYI you can use AWS ES index state management to do this, although it's time-based and not count-based as OP requests

[alxshr]

+1

[samihoda]

+1

[matanbaruch]

+1

[homerobono]

+1

[boosty]

Hi all,

we currently do not plan to add built-in support for this (which might change, of course).

So for now, migrating indexes to UltraWarm must be triggered externally. You just need to be sure not to migrate the active write index, as it would become read-only on UltraWarm.

You can use the Graylog API to find relevant index names, and also to find out if an index is the active write index (is_deflector).

Two options:

1. Create a cronjob to call POST _ultrawarm/migration/my-index/_warm: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ultrawarm.html#ultrawarm-migrating
2. Create an ISM rule (as mentioned above): https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism.html

[tellistone]

I think we should push this up.