[BUG] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://localhost:9200/_bulk"}

[sultanovich]

Describe the bug When I try to send logstash OSS events to elastic I get the following error:

[2023-02-03T02:22:23,881][ERROR][logstash.outputs.elasticsearch][main][0b55d7846aa836aa6ab591125c3a16163b6a5fb910bc6b218f089797b2c2d4f1] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"https://wazuh-indexer:9200/_bulk"}

This error occurs after upgrading from opensearch 1.3.0 to 2.4.1

To Reproduce Steps to reproduce the behavior:

1. Start a logstash pod with the following data

• Dockerfile

FROM docker.elastic.co/logstash/logstash-oss:7.16.2

• Configmap (pipeline)

    input {
        beats {
            port => 5000
        }
    }
    filter {
        json {
          source => "message"
        }
    }
    output {
        elasticsearch {
            hosts => ["https://localhost:9200"]
            ilm_enabled => false
            index => "%{@index_name}"
            user => "${LOGSTASH_KS_USER}"
            password => "${LOGSTASH_KS_PASS}"
            ssl => true
            ssl_certificate_verification => true
            cacert => "/usr/share/logstash/config/ca-chain.cert.pem"
        }
        stdout { codec => rubydebug }
    }

2. Start a pod with elasticsearch opensearch 1.3.0.
3. Send an event to logstash
4. View error in logstash logs.

[2023-02-07T15:13:19,164][ERROR][logstash.outputs.elasticsearch][main][4fa2049b909d01b7125bffcde8b40504c18ebda16a870126c55093583efa9846] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"https://localhost:9200/_bulk"}

Expected behavior The expected behavior is that the event is correctly sent to elastic as it happened with opensearch 1.3.

Plugins Logstash The logstash image includes a lot of plugins, but only logstash-output-elasticsearch is used.

[root@logstash-oss-86c7bf9485-4b6pv logstash]# grep logstash-output-elasticsearch Gemfile
gem "logstash-output-elasticsearch", ">= 10.4.2"
[root@logstash-oss-86c7bf9485-4b6pv logstash]# 
[root@logstash-oss-86c7bf9485-4b6pv logstash]# logstash-plugin list --installed --verbose --group output | grep logstash-output-elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
logstash-output-elasticsearch (10.7.3)
[root@logstash-oss-86c7bf9485-4b6pv logstash]# 

[root@logstash-oss-86c7bf9485-4b6pv logstash]# logstash-plugin list
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
logstash-codec-avro
logstash-codec-cef
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-aggregate
logstash-filter-anonymize
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-de_dot
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop
logstash-filter-elasticsearch
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-http
logstash-filter-json
logstash-filter-kv
logstash-filter-memcached
logstash-filter-metrics
logstash-filter-mutate
logstash-filter-prune
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-translate
logstash-filter-truncate
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-azure_event_hubs
logstash-input-beats
logstash-input-couchdb_changes
logstash-input-dead_letter_queue
logstash-input-elasticsearch
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-jms
logstash-input-pipe
logstash-input-redis
logstash-input-s3
logstash-input-snmp
logstash-input-snmptrap
logstash-input-sqs
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-integration-jdbc
 ├── logstash-input-jdbc
 ├── logstash-filter-jdbc_streaming
 └── logstash-filter-jdbc_static
logstash-integration-kafka
 ├── logstash-input-kafka
 └── logstash-output-kafka
logstash-integration-rabbitmq
 ├── logstash-input-rabbitmq
 └── logstash-output-rabbitmq
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elastic_app_search
logstash-output-elasticsearch
logstash-output-email
logstash-output-file
logstash-output-graphite
logstash-output-http
logstash-output-lumberjack
logstash-output-nagios
logstash-output-null
logstash-output-pipe
logstash-output-redis
logstash-output-s3
logstash-output-sns
logstash-output-sqs
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-webhdfs
logstash-patterns-core
[root@logstash-oss-86c7bf9485-4b6pv logstash]# 

Plugins Opensearch

opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-sql
repository-s3

Host/Environment (please complete the following information):

• OS: CentOS
• Version 7.9.2009

Additional context The error occurs when deploying in EKS the new version of Wazuh v4.4. You can replicate the environment using the Wazuh documentation and adding the logstash component to send events to Wazuh-indexer.

git clone https://github.com/wazuh/wazuh-kubernetes.git -b 4.4 --depth=1
cd wazuh-kubernetes

[dblock]

Does the 2.4.1 come up? Is there another error in the logs?

[sultanovich]

Hi @dblock. No, but upon investigation, we have come to the conclusion that it is because of this breaking change on Opensearch 2.x.

The question here then would be, is there any version of logstash OSS with the logstash-output-elasticsearch plugin that works on Opensearch 2.4.1?

[dblock]

Yes, it's https://rubygems.org/gems/logstash-output-opensearch. Does this work?

[sultanovich]

hi @dblock , just what we see is that with logstash-output-elasticsearch plugin in version 10.7.3 it does not work (I have updated the main comment with this information).

So the question is, is there any version of the logstash-output-elasticsearch plugin that works with Opensearch 2.4.1?

[dblock]

AFAIK no, but you can always ask folks that maintain logstash-output-elasticsearch.

We are actively maintaining logstash-output-opensearch. The errors above in the issue are all -elasticsearch, I don't clearly see an error with logstash-output-opensearch when trying to ingest data. Help narrow down the problem with logstash-output-elasticsearch and let's fix it? I'll move this issue into that project.

[sultanovich]

I understand what you are saying @dblock, the problem here is that we are still with one foot in each product. Surely when I ask the maintainer of logstash-output-elasticsearch which version works correctly on Opensearch 2.x he will tell me that since it is a different product he does not have that information. That is why I was asking here if you have information to confirm if any version higher than 10.7.3 of logstash-output-elasticsearch supports Opensearch 2.x.

For us, it is clear that the definitive solution is to replace the plugin and start using logstash-output-opensearch.

[dblock]

I understand what you are saying @dblock, the problem here is that we are still with one foot in each product. Surely when I ask the maintainer of logstash-output-elasticsearch which version works correctly on Opensearch 2.x he will tell me that since it is a different product he does not have that information. That is why I was asking here if you have information to confirm if any version higher than 10.7.3 of logstash-output-elasticsearch supports Opensearch 2.x.

I am 99% sure that there's no version of logstash-output-elasticsearch that supports OpenSearch 2.x.

[sultanovich]

I understand, thank you anyway for your help @dblock .

[dlvenable]

@sultanovich ,

In order to send data to an OpenSearch cluster from Logstash, you will need to use the logstash-output-opensearch plugin. The latest version of this plugin supports OpenSearch 1.x and 2.x.

You should be able modify your Dockerfile to include a command to run the following command.

bin/logstash-plugin install logstash-output-opensearch

And you can then change your output configuration to use the opensearch plugin instead of elasticsearch.

output {
        opensearch {
            hosts => ["https://localhost:9200"]
            index => "%{@index_name}"
            user => "${LOGSTASH_KS_USER}"
            password => "${LOGSTASH_KS_PASS}"
            ssl => true
            ssl_certificate_verification => true
            cacert => "/usr/share/logstash/config/ca-chain.cert.pem"
        }
        stdout { codec => rubydebug }
    }

[sultanovich]

Gracias @dlvenable ,

That is exactly what we ended up doing. What we wanted to confirm as I mentioned before, before proceeding to replace the plugin used, is if it was possible to get the same result temporarily with some higher version of logstash-output-elasticsearch to avoid a change in production that we would like to test further.