out_opensearch: data stream compatibility in fluent-bit

[adiforluls]

Addresses #7079

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

• [x] Example configuration file for the change
• [x] Debug log output from testing the change

• [ ] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [ ] Run local packaging test showing all targets (including any new ones) build.
• [ ] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [ ] Documentation required for this feature

Backporting

• [ ] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[adiforluls]

Sample configuration file

[Output]
    Name    opensearch
    Match    kube.*
    Host    foo.bar
    Port    9200
    Suppress_Type_Name    true
    Data_Stream_Mode    true
    Data_Stream_Name    test
    Template_File    /fluent-bit/etc/template.json

Debug logs from output

[2023/05/12 06:16:41] [debug] [output:opensearch:opensearch.0] create index template=0
[2023/05/12 06:16:41] [debug] [output:opensearch:opensearch.0] HTTP Status=200 URI=/_bulk
[2023/05/12 06:16:41] [debug] [output:opensearch:opensearch.0] OpenSearch response

[adiforluls]

Inspired by and loosely based on https://github.com/fluent/fluent-plugin-opensearch#configuration---opensearch-output-data-stream

I'll be looking to add support for creating a default index template such that supplying an index template isn't necessary as a follow up if that's okay.

[adiforluls]

Let me know the process for documentation changes as well (I see the docs required label, if I need to submit those changes hand in hand or after this PR goes in)

[adiforluls]

Further looping in @cosmo0920 who I see wrote the fluentd plugin

[leonardo-albertovich]

@cosmo0920 will make an primary review of the plugin focusing on the logical and practical viewpoint as time permits and I will do a secondary review focusing on coding style and lower level bugs once that's done.

This process will take place within the following week.

Thank you very much for taking the time to make this contribution.

[leonardo-albertovich]

Just to add to what @cosmo0920 said, converting those tabs to white spaces is not just better, it's necessary to take this forward.

[adiforluls]

Debug logs

[2023/05/16 09:56:23] [debug] [output:opensearch:opensearch.0] template with given name foo exists
[2023/05/16 09:56:23] [debug] [output:opensearch:opensearch.0] create index template=0
[2023/05/16 09:56:23] [debug] [output:opensearch:opensearch.0] HTTP Status=200 URI=/_bulk

[adiforluls]

cc: @cosmo0920 @leonardo-albertovich

[adiforluls]

I confirmed that the checking for the existence of template file is nice. However, still indentation issue is persist. Could you use 4-spaces indentation instead of 8-spaces when you use white spaces?

Sure, it's my IDE's doing, will fix shortly

[adiforluls]

@cosmo0920 @leonardo-albertovich I think I've addressed all the comments and fixed indentation issues. Let me know if more changes are needed.

Once this goes in, I'll try and squeeze in the logic to create a default index template if the template file isn't provided as a follow up to this.

[adiforluls]

@cosmo0920 forgot that I had edited other files and they also had weird indents. All should be fixed now.

[adiforluls]

@leonardo-albertovich I've addressed all the comments and made the changes requested.

[leonardo-albertovich]

Blessed be thy PR! This is ready to be merged @edsiper

[leonardo-albertovich]

Thanks a lot for the contribution and cooperation @adiforluls!

[adiforluls]

Can we get this merged as soon as possible (possibly today)? It's an important feature for people who're using opensearch output so it'll be good to have this in a release. @edsiper

[adiforluls]

Can a code owner look into this? I have two approvals from maintainers and need code-owner approval to merge. @edsiper or @PettitWesley

[adiforluls]

This PR was approved 3 weeks ago but is still pending merge, let me know if I'm missing some process cc: @edsiper @PettitWesley @leonardo-albertovich @cosmo0920

[PettitWesley]

@adiforluls I think the failing PR check unit test may be a blocker

[adiforluls]

I see that only this check failed https://github.com/fluent/fluent-bit/actions/runs/5023119764/jobs/9008358596?pr=7371 and it doesn't look like the test ran? @PettitWesley or @leonardo-albertovich if you know what this test does and if it's related

[adiforluls]

@leonardo-albertovich or @cosmo0920 would you know if this failing test might be related? To me it looks like it did not even run so it is some unrelated regression. This PR is pending merge for about a month now so in case there's an issue I'd like to fix it ASAP, if not then would want to see this getting merged.

[adiforluls]

Hi @edsiper, I received the approvals from the maintainers 4 months ago but this was never merged. Let me know what's holding up the merge. I've been running this change for months without any issues.

[ng-bsy]

Also pretty much interested in this PR @edsiper @PettitWesley could you please find it in your time to give the needed feedback for moving on? Thanks a lot

[ng-bsy]

@PettitWesley @edsiper Will this PR get your attention some day? Data streams are a pretty fundamental feature within OpenSearch.

[qxmips]

waiting to this functionality

[ecerulm]

@adiforluls , the logs for the CI/CD jobs expired, could you retrigger them?

[ecerulm]

Closes #8522 Closes #7079

[ThoreKr]

This turned out to be a blocker for us for moving from fluentd to fluent-bit. Is there any progress here or alternatives to work with datastreams without a lot of manual interaction?

[krushik]

@ThoreKr, you can set up an index template with the ‘Template type’ set to ‘Data streams’ and the required index pattern pre-configured on the OpenSearch side. Then you can use the regular write_operation 'create' in Fluent Bit with any index name that falls within that index pattern. This approach has been working well for us