Feature request: add support the output to multiple elasticsearch hosts or cluster

[jenningsloy318]

Currently fluent-bit can only set only one elasticsearch instance as the output, but actually cluster setup is common in elasticsearch, so hope we can add loadbalance mechnism to export the logs to muliple elasticseach cluster/instance.

something like this:

[OUTPUT]
    Name  es
    Match *
    Host  192.168.2.3,192.168.2.4,
    Port  9200
    Index my_index
    Type  my_type

[emretiryaki]

is there any development for this feature ?

[zpDeveloper]

I have the same problem. Will it be solved?

[agup006]

Replying on an old thread, If I understand correctly, the Upstream featureset should allow the definition of multiple Elasticsearch nodes and the ability to rotate amongst them when sending data.

https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/upstream-servers

Closing for now, feel free to re-open if needed

[mrajcok]

Replying on an old thread, If I understand correctly, the Upstream featureset should allow the definition of multiple Elasticsearch nodes and the ability to rotate amongst them when sending data. https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/upstream-servers

@agup006 The Elasticsearch output plugin has no Upstream parameter, so how would this work?

[mrajcok]

I would like this issue or #1340 reopened. I.e., to add upstream server support to the Elasticsearch output plugin. This would allow each fluent-bit container to balance across multiple Elasticsearch ingest nodes (that are in the same Elasticsearch cluster).

[thatsk]

is any update on this. ?

[varuzam]

it would be really nice feature

[thatsk]

why this closed state it should be open ?

[awaizman1]

hi,

elasticsearch output plugin doesn't support upstream. Can you please consider modifying the ES plugin to be able to get a list of hosts instead of one? similar to fluentd capability https://docs.fluentd.org/output/elasticsearch#hosts-optional

[lecaros]

Hello, Have you configured your upstream with its nodes and then add it to the ES plugin? Are you seeing any errors? Can you share your config?

[kunalsingthakur]

yes its not identifying multiple host in host section https://github.com/fluent/fluent-bit/issues/664#issue-336921045

[lecaros]

You have to use upstream in this case. https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/upstream-servers

[awaizman1]

hi @lecaros,

tried to configure an upstream and provide it to the ES output plugin. seems like the plugin doesn't support upstream, this is the error I get: [2023/10/18 08:49:36] [error] [config] es: unknown configuration property 'upstream'. The following properties are allowed: index, type, suppress_type_name, http_user, http_passwd, compress, cloud_id, cloud_auth, aws_auth, aws_region, aws_sts_endpoint, aws_role_arn, aws_external_id, logstash_format, logstash_prefix, logstash_prefix_key, logstash_dateformat, time_key, time_key_format, time_key_nanos, include_tag_key, tag_key, buffer_size, path, pipeline, generate_id, write_operation, id_key, replace_dots, current_time_index, trace_output, and trace_error.

configuration used:

[UPSTREAM]
  es_cluster
    
[NODE]
  name     node-1
  host       10.200.56.111

and in output section:

[OUTPUT]
  Name es
  Alias SendApplicationLog
  Match kube.*
  Upstream es_cluster
  Index fluentbit

[awaizman1]

Hi @lecaros Did you have a chance to check this? As mentioned ES output plugin doesn’t support upstream

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[awaizman1]

Hi

is there another workaround for the use case of having es cluster and make sure fluentbit will succeed to emit records in case one node is down?