[BUG] OpenSearch - Filebeat: enable sample dashboards

[rafzei]

Describe the bug Filebeat comes packaged with example Kibana dashboards, visualizations, and searches for visualizing Filebeat data in Kibana. After migration to OpenSearch this feature stops working. Sevice won't start. The error:

filebeat[32347]: 2022-05-17T00:17:24.417Z        ERROR        instance/beat.go:971        Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://...

Currently, the flag is set to false: setup.dashboards.enabled: false

How to reproduce Steps to reproduce the behavior:

1. execute epicli init ... (with params)
2. edit config file
3. execute epicli apply ...

Expected behavior A clear and concise description of what you expected to happen.

Config files If applicable, add config files to help explain your problem.

Environment

• Cloud provider: [AWS | Azure | All | None]
• OS: [e.g. Ubuntu 20.04.3 LTS, you can use cat /etc/os-release]

epicli version: [epicli --version]

Additional context

This is probably happening due to a version mismatch. Some details could be found there: https://www.electricbrain.com.au/pages/analytics/opensearch-vs-elasticsearch.php

---

DoD checklist

• Changelog
  • [ ] updated
  • [ ] not needed
• COMPONENTS.md
  • [ ] updated
  • [ ] not needed
• Schema
  • [ ] updated
  • [ ] not needed
• Backport tasks
  • [ ] created
  • [ ] not needed
• Documentation
  • [ ] added
  • [ ] updated
  • [ ] not needed
• [ ] Feature has automated tests
• [ ] Automated tests passed (QA pipelines)
  • [ ] apply
  • [ ] upgrade
  • [ ] backup/restore
• [ ] Idempotency tested
• [ ] All conversations in PR resolved

[cicharka]

Summary from the current research

The main issue blocking ultimate success are the version checks carried out prior to any attempt to load stuff. Essentially, these checks have to be bypassed.

• Work done based on article: https://www.electricbrain.com.au/pages/analytics/opensearch-vs-elasticsearch.php which comes from the BUG: https://github.com/opensearch-project/OpenSearch-Dashboards/issues/656
• Tested with versions of OpenSearch and Filebeat installed by defaul by Epiphany, as well as versions used in article mentioned above.

Steps to follow:

1. Manually load the elastic template in to OpenSearch
   1. This is the step that is causing problems and most probably is the key point to focus on.
2. Manually load an index-pattern in to Opensearch-Dashboards
3. Manually load a dashboard in to Opensearch-Dashboards

Step 1 -> Manually load the elastic template in to OpenSearch

See https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html for more information about loading templates.

1. Produce the artefacts needed by the upload process

sudo filebeat export template > filebeat.template.json

2. Modify template by adding in settings a section for incompatible 'lifecycle':

# part of filebeat.template.json
...
  "settings": {
    "index": {
      "lifecycle": {
        "name": "filebeat",
        "rollover_alias": "filebeat-7.12.1"
      },
      "mapping": {
        "total_fields": {
          "limit": 10000
        }
      },
...

2. Load the template into the OpenSearch

sudo curl -k -XPUT -H 'Content-Type: application/json' \
https://cicharka-filebeater-logging-vm-0:9200/_template/filebeat-7.12.1 \
[email protected] \
-u 'admin:PASSWORD_TO_CHANGE' | python3 -mjson.tool

Unfortunately this steps doesn't succeed -> loading template returns response with error:

"type": "illegal_argument_exception",
                "reason": "unknown setting [index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"

In order to give a try other way, we can modify /etc/filebeat/filebeat.yml. Inside that file there's a section for setup which contain part Index Lifecycle Management (ILM):

# ====================== Index Lifecycle Management (ILM) ======================

# Configure index lifecycle management (ILM). These settings create a write
# alias and add additional settings to the index template. When ILM is enabled,
# output.elasticsearch.index is ignored, and the write alias is used to set the
# index name.

# Enable ILM support. Valid values are true, false, and auto. When set to auto
# (the default), the Beat uses index lifecycle management when it connects to a
# cluster that supports ILM; otherwise, it creates daily indices.
# Disabled because ILM is not enabled by default in Epiphany
setup.ilm.enabled: false

# Set the prefix used in the index lifecycle write alias name. The default alias
# name is 'filebeat-%{[agent.version]}'.
#setup.ilm.rollover_alias: 'filebeat'

# Set the rollover index pattern. The default is "%{now/d}-000001".
#setup.ilm.pattern: "{now/d}-000001"

# Set the lifecycle policy name. The default policy name is
# 'beatname'.
#setup.ilm.policy_name: "mypolicy"

Modified file with uncommented and adjusted fields:

...
setup.ilm.enabled: true
setup.ilm.rollover_alias: 'filebeat-%{[agent.version]}'
setup.ilm.policy_name: "filebeater"
...

Restarting filebeat with systemctl restart filebeat.service results in an error seen in logs:

Failed to connect to backoff(elasticsearch(https://cicharka-filebeater-logging-vm-0:9200)): Connection marked as failed because the onConnect callback failed: ILM is not supported by the Elasticsearch version in use

Next steps that should be executed after Step 1 succeeds.

Step 2 -> Manually load an index-pattern in to Opensearch-Dashboards

1. Produce the artefacts needed by the upload process (index-pattern)

filebeat export index-pattern --es.version 7.12.1 > filebeat.index-pattern.json

2. Import index-pattern to OpenSeachDashboards

curl -XPOST -u 'admin:PASSWORD_TO_CHANGE' \
-H 'osd-xsrf: true' \
-H 'Content-Type: application/json' \
http://cicharka-filebeater-logging-vm-0:5601/api/opensearch-dashboards/dashboards/import?force=true \
[email protected] -k

Step 3 - > Manually load a dashboard in to Opensearch-Dashboards

curl -XPOST -u 'admin:PASSWORD_TO_CHANGE' \
-H 'osd-xsrf: true' \
-H 'Content-Type: application/json' \
'http://cicharka-filebeater-logging-vm-0:5601/api/opensearch-dashboards/dashboards/import?exclude=index-pattern&force=true' \
-d@/usr/share/filebeat/kibana/7/dashboard/Filebeat-syslog.json

Note that even if the Step 1 fails, we can use step 2 and step 3 in order to load dashboards (although they will have problems to load).