Create External ELK for Monitoring

[punitmundra]

Create Elastic search for the logs consumption from Journal beat and Metrics beat (Both for exisiting and AWS) - Along with curator and Kibana packaging.

Description

This must be config driven and in case user wants us to setup elastic search, they should enable this in a2ha.rb file and accordingly we can create a new set of ES for our LMM purpose. (Note:- we can reuse existing terraform script ).

Should take security group and IAM role and policies in consideration while adding this piece of terraform script.

We should also add curator and kibana services along with ES using terraform. On completion of this story, will raise PR to move the terraform code based to automate.

Acceptance Criteria

Terraform should spun up a new ES cluster/VM as per requirement to accommodate LMM needs, in case customer select and enable this configuration.

Assumption / Remark

Customer may chose to configure their own esiting ES or the ES which we have created for them(if they choose that option).

In total there will be two ES, one for automate data (old one) and the other for LMM (in case user likes us to create the ES)

Aha! Link: https://chef.aha.io/epics/SH-E-643

[jayvikramsharma1]

Hi

[jayvikramsharma1]

Checklists

• read config for LMM = true/false and elk = true/false
• based on setting if LMM is true and elk = true need to read instance size for ELK like "m5a.large" or "t3a.medium" and no nodes for elasticsearch
• write terraform script to create security group and subnets for elk stack
• Write terraform script to launch elasticsearch in AWS with new subnets and security group
• install elasticsearch on each instance and configure all instances.
• Write terraform script to launch Kibana instance.
• write terraform script to install Kibana on instance.
• terraform script to configure kibana with elasticsearch nodes.
• test case to verify all nodes are well configured with kibana.
• change config for metricbeat and journal beat to push logs in new elk stack.
• test case to verify logs from all journal beat and metricbeat are getting recorded in new elk stack

[jayvikramsharma1]

Look like I don't have proper permission to move tickets and create checklists or subtasks in Zenhub. I am continuously getting error like "check your permission"

[jayvikramsharma1]

• Based on config available for LMM

• in case of logging_and_monotoring = true

• and new_elk = true

• it will be going to deploy a new AWS ELK stack for LMM and It will send existing metricbeat and journalbeat logs to new LMM ELK stack

https://github.com/chef/a2-ha-backend/pull/493