Thibault Richard

> If you don't set these up with persistence, then every time you recreate them, the states are lost and start over. ECK uses an `hostPath` volume to create a...

Indeed, `hostpath` does not work when an Agent Pod is re-scheduled on another k8s node. Makes sense, thanks for the explanation.

Relates to https://github.com/elastic/cloud-on-k8s/issues/2448.

Closing because it seems to be a temporary problem, which has not happened again for over a year.

Closing because similar to #5795.

@aravindh-odyssey, as mentioned in https://github.com/elastic/cloud-on-k8s/issues/3598#issuecomment-1028268629, Elastic supports an official [Elastic Stack Terraform provider](https://github.com/elastic/terraform-provider-elasticstack) that allows you to manage and configure the Elastic Stack as code using Terraform.

Closing because it is a transient problem that has not reoccurred for almost 1 year.

Related to: - https://github.com/elastic/kibana/issues/131505

In [build/4899](https://buildkite.com/elastic/cloud-on-k8s-operator/builds/4899#01894b31-1fcf-423f-8694-8ccc591020f3), there was exactly 10 x `Elasticsearch cluster health check failure at ...: cluster health red`, and in #7000 we set the limit to `10`, so it should have...

Despite #7012, we had a new recent failure. https://buildkite.com/elastic/cloud-on-k8s-operator-nightly/builds/324 ``` === RUN TestAgentVersionUpgradeToLatest8x/Elasticsearch_cluster_health_should_not_have_been_red_during_mutation_process steps_mutation.go:160: Elasticsearch cluster health check failure at 2023-10-06 23:35:43.223407119 +0000 UTC m=+2275.455997811: cluster health red steps_mutation.go:160: Elasticsearch...