elk 实践

[heidsoft]

查看索引

➜  elasticsearch-7.8.1 curl -XGET 'localhost:9200/_cat/indices'
yellow open metricbeat-7.8.1-2020.08.11-000001 Rk__8HYyQWeM0d9xfl6wZA 1 1 49164   0 13.1mb 13.1mb
green  open .apm-custom-link                   zyZJQAw0Q0Glxa6OtaRS8g 1 0     0   0   208b   208b
green  open .kibana_task_manager_1             kRW3wcm_RXmwD_hGOSV9jw 1 0     5  10 10.8kb 10.8kb
green  open .apm-agent-configuration           4_4_VFGAQoGTwx0fig8xHg 1 0     0   0   208b   208b
green  open kibana_sample_data_logs            EGSClSsPSIGHlnDkjxqraQ 1 0 14074   0   11mb   11mb
green  open .async-search                      6A8gMDJASlqMYsB51s451w 1 0     3   0  6.8mb  6.8mb
green  open .kibana_1                          fdk86yC0RtO48RpI4hH6gw 1 0  3176 164  2.1mb  2.1mb
green  open .kibana-event-log-7.8.1-000001     ab62kehLQlOFjRJPBPrUQg 1 0     2   0 10.4kb 10.4kb
yellow open filebeat-7.8.1-2020.08.11-000001   LRoh9ocGSVmc6PdHnPNd6g 1 1 73091   0 12.9mb 12.9mb

➜ elasticsearch-7.8.1

分片状态查看

➜  elasticsearch-7.8.1 curl -XGET 'localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned'
.apm-custom-link                   0 p STARTED
.apm-agent-configuration           0 p STARTED
.async-search                      0 p STARTED
metricbeat-7.8.1-2020.08.11-000001 0 p STARTED
metricbeat-7.8.1-2020.08.11-000001 0 r UNASSIGNED
kibana_sample_data_logs            0 p STARTED
filebeat-7.8.1-2020.08.11-000001   0 p STARTED
filebeat-7.8.1-2020.08.11-000001   0 r UNASSIGNED
.kibana_1                          0 p STARTED
.kibana_task_manager_1             0 p STARTED
ilm-history-2-000001               0 p STARTED
.kibana-event-log-7.8.1-000001     0 p STARTED
➜  elasticsearch-7.8.1

ELK实现日志监控告警

https://www.ctolib.com/docs/sfile/ELKstack-guide-cn/elasticsearch/other/elastalert.html https://anjia0532.github.io/2017/02/14/elasticsearch-elastalert/ https://media.readthedocs.org/pdf/elastalert/latest/elastalert.pdf https://anjia0532.github.io/2017/02/14/elasticsearch-elastalert/ https://blog.51cto.com/seekerwolf/2121070 https://www.ctolib.com/docs/sfile/ELKstack-guide-cn/elasticsearch/other/elastalert.html https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html

[heidsoft]

logstash-plugin

➜ logstash-6.6.0 bin/logstash-plugin install logstash-input-beats Validating logstash-input-beats Installing logstash-input-beats Installation successful ➜ logstash-6.6.0 bin/logstash-plugin install logstash-codec-rubydebug Validating logstash-codec-rubydebug Installing logstash-codec-rubydebug Installation successful

[heidsoft]

性能优化

https://www.datadoghq.com/blog/elasticsearch-performance-scaling-problems/

[heidsoft]

ElasticSearch 状态查询Cat接口

https://yq.aliyun.com/articles/676135

[heidsoft]

es 性能优化

CentOS7上安装配置破解Elasticsearch+Kibana 6.4.2-6.5.1全过程

[heidsoft]

Kibana 用户指南（Kibana中的X-Pack设置）

[heidsoft]

es深度分页机制

[heidsoft]

api快速入门

[heidsoft]

Garbage Collection in Elasticsearch and the G1GC Centralizing logs at Naukri.com with Kafka and ELK stack Elasticsearch In Production — Deployment Best Practices Garbage Collection Settings for Elasticsearch Master Nodes

[heidsoft]

ElasticSearch 内部机制浅析（二） ElasticSearch 内部机制浅析（一） 集群内的原理 分片内部原理 » 持久化变更

[heidsoft]

集群监控

https://www.elastic.co/guide/en/elasticsearch/reference/6.3/cluster-stats.html https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-state.html

[heidsoft]

es写入流程

Elasticsearch写入原理深入详解 Elasticsearch 数据写入流程

[heidsoft]

es性能优化

索引性能技巧 超详细的Elasticsearch高性能优化实践 一次ES优化过程 Can I configure my Linux system for more aggressive file system caching? 预加载数据到文件系统缓存 Elasticsearch from the Top Down Elasticsearch写入索引数据的过程 以及优化写入过程 Anatomy of an Elasticsearch Cluster: Part I On ElasticSearch performance The Authoritative Guide to Elasticsearch Search Tuning (Part 3) https://dzone.com/articles/elasticsearch-performance-tuning

[heidsoft]

https://www.acloudtree.com/hacking-java-bytecode-for-programmers-part1-the-birds-and-the-bees-of-hex-editing/

[heidsoft]

es api 测试

curl -X PUT "localhost:9200/test" -H 'Content-Type: application/json' -d'
{
    "settings" : {
        "index" : {
            "number_of_shards" : 1, 
            "number_of_replicas" : 0
        }
    },
    "mappings": {
    	"_doc":{
	    	"properties": { 
			      "title":    { "type": "text"  }, 
			      "name":     { "type": "text"  }, 
			      "age":      { "type": "integer" },
			      "created":  {
			          "type":   "date", 
			          "format": "strict_date_optional_time||epoch_millis"
			        }
		    }
    	}   
   }
}
'

curl -X PUT "localhost:9200/test/_mapping/_doc" -H 'Content-Type: application/json' -d'
{
  "properties": { 
        "title":    { "type": "text"  }, 
        "name":     { "type": "text"  }, 
        "age":      { "type": "integer" },  
        "created":  {
          "type":   "date", 
          "format": "strict_date_optional_time||epoch_millis"
        }
      }
}
'

[heidsoft]

副本机制模型

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-replication.html#docs-replication

[heidsoft]

Elasticsearch 节点选举、分片及 Recovery

https://v2ex.com/t/652785

ElasticSearch分布式原理探究 —— 节点和分片

https://www.mantian.site/blog/2019/09/09/ElasticSearch%E5%88%86%E5%B8%83%E5%BC%8F%E5%8E%9F%E7%90%86%E6%8E%A2%E7%A9%B6-%E2%80%94%E2%80%94-%E8%8A%82%E7%82%B9%E5%92%8C%E5%88%86%E7%89%87/