docker-symfony
docker-symfony copied to clipboard
Unallocated shard in Elasticsearch, and often unavailable
After setting up and running on a Digital Ocean droplet (4 GB RAM) I had multiple problems. Logstash was able to set up an index in Elasticsearch - logstash-2020.12.09-000001
- but Elasticsearch stayed in yellow health status because of an unassigned shard. Querying Elasticsearch with GET /_cat/shards?v
showed
{
"cluster_name": "docker-cluster",
"status": "yellow",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 1,
"active_shards": 1,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 1,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 87.5,
"indices": {
"logstash-2020.12.09-000001": {
"status": "yellow",
"number_of_shards": 1,
"number_of_replicas": 1,
"active_primary_shards": 1,
"active_shards": 1,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 1,
"shards": {
"0": {
"status": "yellow",
"primary_active": true,
"active_shards": 1,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 1
}
}
}
}
}
Notice "unassigned_shards": 1,
- the value should be 0 as this is a single node cluster.
Doing
GET _cluster/allocation/explain -d '{
"index": "logstash-2020.12.09-000001",
"shard": 0,
"primary": true
}'
showed:
{
"index": "logstash-2020.12.09-000001",
"shard": 0,
"primary": true,
"current_state": "started",
"current_node": {
"id": "9d_i9BLtTQGndsrZqY2CIQ",
"name": "a8c5e5ec8685",
"transport_address": "172.20.0.3:9300",
"attributes": {
"ml.machine_memory": "4127453184",
"xpack.installed": "true",
"transform.node": "true",
"ml.max_open_jobs": "20"
},
"weight_ranking": 1
},
"can_remain_on_current_node": "yes",
"can_rebalance_cluster": "no",
"can_rebalance_cluster_decisions": [
{
"decider": "rebalance_only_when_active",
"decision": "NO",
"explanation": "rebalancing is not allowed until all replicas in the cluster are active"
},
{
"decider": "cluster_rebalance",
"decision": "NO",
"explanation": "the cluster has unassigned shards and cluster setting [cluster.routing.allocation.allow_rebalance] is set to [indices_all_active]"
}
],
"can_rebalance_to_other_node": "no",
"rebalance_explanation": "rebalancing is not allowed"
}
I'm not an Elasticsearch expert, and now I'm having to read documentation on what all of this stuff means, and what to do about it. Eventually I did
PUT /_settings
{
"index" : {
"number_of_replicas" : 0
}
}
Ok great! Elasticsearch is now green!
However, logstash just can't talk to Elasticsearch. From the logs:
logstash | [2020-12-09T19:06:00,332][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
What now?! Sshing into the logstash container, I did a number of curl http://elasticsearch:9200/
and only the first one worked:
curl http://elasticsearch:9200/
{
"name" : "a8c5e5ec8685",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "iT6MF_DsTNSOylme1dMgPg",
"version" : {
"number" : "7.9.3",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "c4138e51121ef06a6404866cddc601906fe5c868",
"build_date" : "2020-10-16T10:36:16.141335Z",
"build_snapshot" : false,
"lucene_version" : "8.6.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
bash-4.2$ curl http://elasticsearch:9200/
curl: (7) Failed connect to elasticsearch:9200; Connection refused
bash-4.2$ curl http://elasticsearch:9200/
curl: (7) Failed connect to elasticsearch:9200; Connection refused
bash-4.2$ curl http://elasticsearch:9200/
curl: (7) Failed connect to elasticsearch:9200; Connection refused
bash-4.2$ curl http://elasticsearch:9200/
curl: (7) Failed connect to elasticsearch:9200; Connection refused
bash-4.2$
And Elasticsearch is throwing this exception on boot:
"stacktrace": ["org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][_doc][space:default]: routing [null]]",
It's probably because I only have 4 GB RAM on my Digital Ocean droplet; https://elk-docker.readthedocs.io/#prerequisites says that it needs A minimum of 4GB RAM assigned to Docker
. If this is the case, it should be mentioned in this project's readme.