Create a mechanism that deploys Observability Stack and Logging Stack using OpenSearch

[TheAlgo]

Is your feature request related to a problem? Please describe. Everyone have their own system and tool to have these kind of stacks setup along with the core OpenSearch and OpenSearch Dashboards. It will be awesome to have a standard setup way which can easily self managed and is cloud agnostic at the same time.

Describe the solution you'd like There can be multiple solutions to it:

1. Have a provisioning tool which someone can install on their machine and it can help setup these stacks.
2. An awesome docker compose to help setup this.
3. For K8s maybe have an extension in the OpenSearch Operator to help spin up logging and observability stacks. We can also have a helm chart which takes in dependency from the component charts and setup these stacks.

So if I want to deploy a stack with (OpenSearch + Dashboards + Logstash). I can do them at one go at one shot.

Describe alternatives you've considered There are some blogs around it but nothing that can be standardised. If some standard tools that can be developed. This will help the broader audience and will serve as pretty cool starter for them

Additional context These can be further extended to build intelligent tooling to solve many more such interesting usecases

[TheAlgo]

@dblock @peterzhuamazon @bbarani What do you all think about it?

[dblock]

Are you looking for something like plural.sh? Haven't tried it myself.

[TheAlgo]

Yup something along the lines of it but more to be suitable for setting up OpenSearch specific apps/stacks.

[peterzhuamazon]

Helm repo can create a new chart. Ansible repo can create a new playbook.

I am not sure if we want to maintain multiple copies of the same mechanism, but consider we already maintain distinct copy of chart and playbook, it makes sense to me to create one for each.

[prudhvigodithi]

Logstash with combination of fluentD is used at most places as data collectors, fluentD as an agent captures logs and forwards them to Logstash and Logstash parses the log and pushes to OpenSearch, sometimes Logstash is replaced with kafka or used both in combination for advance usage (with kafka input plugin). This combination of Logstash + fluentD + OpenSearch + OpenSearch-Dashboard would be an ideal logging stack. With Helm we can add dependency charts (using Charts.yaml )that can install already existing Logstash and fluentD helm charts along with OpenSearch. With docker-compose also this can be achieved using depends_on conditions. May be we can start by adding a blog with all these service deployed showing a sample use case to start with.