[3] Make fluentd find the default index of each alias at start

[robinelastisys]

Describe the bug By default, OpenSearch creates one default write_index(e.g. kubernetes-default-2022.08.26-000001) for some of the default aliases we have. If fluentd-plugin-* can't find this default index(in the alias), they try to create new ones.

The big problem: The new index fluentd creates can not be set to be the main write_index as the default index is the main one already. The "auto_create_index" functionality seems to have some limitations to it and doesn't create the new indexes pointed to the right aliases either. Because of this, we don't want to rely on the plugins solely but rather let the ISM(OpenSearch itself) do the work together with fluentd in a combo solution if possible.

End conclusion & solution: Re-configure fluentd to find the default write_index in each alias in the first place instead of trying to create a new one when it cannot be found.

To Reproduce Setup a standard environment with no 'auto_create_index'-functionality, edit the deamonsets 'fluentd-system-fluentd-elasticsearch' and 'fluentd-fluentd-elasticsearch' to 'OUTPUT_LOG_LEVEL: debug' and 'grep' for "index_not_found" or "illegal_argument".

e.g. {"time":"2022-08-22 07:52:29 +0000","level":"debug","message":"[elasticsearch_kubernetes] Indexed (op = index), 10 index_not_found_exception"}

error="400 - Rejected by OpenSearch [error type]: illegal_argument_exception [reason]: 'no write index is defined for alias [kubernetes]

Expected behavior We want fluentd to find the proper indexes by default - furthermore, have OpenSearch itself take care of creating new indexes if some go missing with the right settings. IF "fluentd-plugin-*" can do it, that's also nice, but previous findings suggest it may be problematic.

Version (add all relevant versions):

• Compliant kubernetes apps version 24.1 or newer.

Additional context This issue evolved from - https://github.com/elastisys/compliantkubernetes-apps/issues/1084