out_es: Add target_index variable using record accessor syntax.

[Wiston999]

trafficstars

Add target_index variable for rendering ES index name using record accessor syntax. This allows to define ES index name based on record accessor, enabling to use record field values as destination ES index. It differs from logstash_prefix_key as no time component is added to index name, so ES data streams can be used as target index easily.

Tests output and other information at https://gist.github.com/Wiston999/c7d3ff5389f64fdf1113146090ab1aa1.

Addresses #2514

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

• [x] Example configuration file for the change (here)
• [x] Debug log output from testing the change (30 seconds test execution log and out_elasticsearch run_code_analysis output enabled)

• [x] Attached Valgrind output that shows no leaks or memory corruption was found (here)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [x] Documentation required for this feature

https://github.com/fluent/fluent-bit-docs/pull/1163

Backporting

• [N/A] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[patrick-stephens]

I think for Opensearch, Index was just updated to support RA syntax - would it be better to do that for ES as well rather than a new key?

[Wiston999]

I think for Opensearch, Index was just updated to support RA syntax - would it be better to do that for ES as well rather than a new key?

I considered that option too. I'm open to change to RA on Index and adding a new Index_default or whatever variable name for a fallback index name in case RA doesn't render any value.

Just confirm me if you prefer that option and I'll make the changes.

[Wiston999]

I've updated the PR to use the same approach as out_opensearch plugin, it is, detect if Index variable has $ and treat it like a RA expression.

Updated gist .

[DandyDeveloper]

Anybody looking at this? I'm also impacted and seems as though opensearch as an alternative broke with the API changes in recent release, so I can't use that as a bandaid.

👍

[braydonk]

What happened with the commits on this PR? Whatever happened last week, where it added a bunch of commits from master, should probably be resolved. Only the 3 out_es commits that are actually part of this PR should be here.

[Wiston999]

What happened with the commits on this PR? Whatever happened last week, where it added a bunch of commits from master, should probably be resolved. Only the 3 out_es commits that are actually part of this PR should be here.

I've just fixed the mess with commits from master. I'm sorry for the noise introduced in the PR and for the inconveniences to other committers.

[Wiston999]

Hi @braydonk , is there anything pending on my side? Failed unit tests are related to plugins that I've not modified and I've also seen that they are failing on master branch.

[braydonk]

I'm sorry, I'm not able to review this PR. I was just pinged as a codeowner of a separate plugin when the extra commits were here. I don't know anything about ES.

[DandyDeveloper]

@Wiston999 Any chance you can pull in origin/master? I can't review, but I'd like to try building from your fork for the time being as I'm stuck on this problem as well.

[Wiston999]

@Wiston999 Any chance you can pull in origin/master? I can't review, but I'd like to try building from your fork for the time being as I'm stuck on this problem as well.

I'm sorry, but I'm not able to merge anything into master. I'm just waiting for some code owner or maintainer to give some attention to this PR.

In the meanwhile we have been using opensearch output plugin that already includes this functionality in our environment (elasticsearch 7.17 & fluent-bit 2.1.8).

[DandyDeveloper]

@Wiston999 To be clear, I meant pulling in origin to your fork, so it's the latest code.

Yeah, the opensearch plugin doesn't work with the latest ES builds unfortunately. Bulk requests are throwing errors.

[DandyDeveloper]

@Wiston999

[2023/08/29 13:22:37] [ info] [fluent bit] version=2.1.9, commit=6530852f3e, pid=1
[2023/08/29 13:22:37] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/08/29 13:22:37] [ info] [cmetrics] version=0.6.3
[2023/08/29 13:22:37] [ info] [ctraces ] version=0.3.1
[2023/08/29 13:22:37] [ info] [input:kafka:kafka.0] initializing
[2023/08/29 13:22:37] [ info] [input:kafka:kafka.0] storage_strategy='memory' (memory only)
[2023/08/29 13:22:37] [engine] caught signal (SIGSEGV)
#0  0x7fd499c0cab8      in  ???() at 4/multiarch/strlen-evex.S:77
#1  0x556670fbd58e      in  flb_strdup() at include/fluent-bit/flb_str.h:46
#2  0x556670fc127f      in  flb_upstream_create() at src/flb_upstream.c:286
#3  0x5566712f5712      in  flb_es_conf_create() at plugins/out_es/es_conf.c:230
#4  0x5566712eaef8      in  cb_es_init() at plugins/out_es/es.c:658
#5  0x556670f7ce31      in  flb_output_init_all() at src/flb_output.c:1301
#6  0x556670fa122e      in  flb_engine_start() at src/flb_engine.c:787
#7  0x556670f4526a      in  flb_lib_worker() at src/flb_lib.c:638
#8  0x7fd49a2e2ea6      in  start_thread() at reate.c:477
#9  0x7fd499b96a2e      in  ???() at sysv/linux/x86_64/clone.S:95
#10 0xffffffffffffffff  in  ???() at ???:0

Getting this segfault when using this kind of output using your branch + master merged

      [OUTPUT]
          Name                es
          Match               *
          Host                ${FLUENT_ELASTICSEARCH_HOST}
          Port                ${FLUENT_ELASTICSEARCH_PORT}
          HTTP_User           ${FLUENT_ELASTICSEARCH_USERNAME}
          HTTP_Passwd         ${FLUENT_ELASTICSEARCH_PASSWORD}
          Id_Key              id
          Index               $topic

Any ideas if something you've changed broke this?

[Wiston999]

Hi @DandyDeveloper , the line of code throwing the seg fault seems https://github.com/fluent/fluent-bit/blob/master/src/flb_upstream.c#L286 , which seems to be related to some proxy configuration.

[DandyDeveloper]

Any ideas on when this might get merged in?

[DandyDeveloper]

@edsiper @leonardo-albertovich Can we get some eyes on this please?

[ajax-bychenok-y]

@DandyDeveloper I came up with next idea, maybe it will be useful for you. It's clusteroutput for K8s fluentbit-operator resource but I think it's quite easily can be transformed to just daemon config (check spec section):

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  annotations:
    meta.helm.sh/release-name: fluent-operator
    meta.helm.sh/release-namespace: fluent
  labels:
    app.kubernetes.io/managed-by: Terraform
    fluentbit.fluent.io/component: logging
    fluentbit.fluent.io/enabled: "true"
  name: es
spec:
  es:
    bufferSize: 50MB
    generateID: true
    host: ${output_es_host}
    # That's a hack for not creating datastreams with date suffix
    logstashDateFormat: eks
    logstashFormat: true
    # In this index go all records w/o proper label 
    logstashPrefix: company-unknown
    logstashPrefixKey: kubernetes['labels']['app.kubernetes.io/name']
    port: 9200
    replaceDots: false
    suppressTypeName: "On"
    timeKey: '@timestamp'
  matchRegex: (?:kube|service)\.(.*)

So fluentbit sends everything in kubernetes['labels']['app.kubernetes.io/name'] datastream name (index template with datastream creating option needs to be previously turned on) and adds eks suffix in the end. Yes, it's quite hacky.

[DandyDeveloper]

@ajax-bychenok-y Thanks for the suggestion! This could work in theory actually, I'll have a crack. But, I'm a little shocked this PR has been pending for so long... It's a pretty simple addition.

[Wiston999]

Hi @edsiper or any other maintainer. Are these changes being considered to be merged? If so, is there any pending action from my side to be taken that is preventing the PR to be merged?

We would like to start upgrading our ES clusters to v8 and as @DandyDeveloper pointed out these changes are needed to have index RA behavior on ES v8 clusters.

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[DandyDeveloper]

Anyone alive

[2nfree]

Is anyone else reviewing or keeping track of this commit?

[cw-Guo]

Hi @cosmo0920 @edsiper , is there anything else needed to do in this pr before it can be merged?

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[eschabell]

@Wiston999 would you mind addressing the conflicts and @patrick-stephens can you look at this for a review?