kafka-connect-elasticsearch
kafka-connect-elasticsearch copied to clipboard
confluentinc
Kafka Connect Elasticsearch connector
I am using elastic search sink connector in distributed mode(2 instance). With task of 8 and about 20 to 25 topics to be sink’ed to elastic search. Even when there...
Hi, I keep getting the error **"WARN || Ignoring version conflict for operation INDEX on document version -1 in index"** on one of my topics. I use the same settings...
### TL;DR Elasticsearch becomes unavailable in the middle of a connection, which makes read.timeout.ms be exceeded. I expected the task to fail, but the task keeps the status `RUNNING`. ###...
Hey, We are using an kafka connector elasticsearch, with the following configuration: `connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector type.name=_doc transforms.insertRecordTimestamp.timestamp.field=@timestamp tasks.max=2 transforms.convertRecordTimestamp.target.type=string transforms.convertRecordTimestamp.type=org.apache.kafka.connect.transforms.TimestampConverter$Value connection.timeout.ms=50000 transforms=routeTS,insertRecordTimestamp,convertRecordTimestamp transforms.routeTS.topic.format=-${timestamp}-s max.retries=1000 key.ignore=true retry.backoff.ms=30000 max.buffered.records=100000 read.timeout.ms=10000 topics= transforms.routeTS.type=org.apache.kafka.connect.transforms.TimestampRouter batch.size=10000 transforms.convertRecordTimestamp.format=yyyy-MM-dd'T'HH:mm:ss.SSSZ...
Hey, I deploy my connector as docker image to Kubernetes. When I set `CONNECT_LOG4J_ROOT_LOGLEVEL: ERROR` the connector throws an error: ``` ERROR StatusLogger Log4j2 could not find a logging implementation....
We used kafka-connect to sync data from kafka to elasticsearch cluster version used : 5.4.1 In test enviroment,we setup es cluster with 3 nodes suddenly a node was crashed,but there...
We bumped into this issue by accident, assuming that we were passing through a null-keyed message. The connector code - possibly in combination with the configuration property _drop.invalid.message_ - takes...
Kafka Connect for Elastic search fails to publish few messages out of 10million messages to Elastic
Hi there, We have a Kafka connect set up for sinking the events from Kafka to Elastic. We are experiencing data consistency issue and was hoping to get some guidance...
## Problem Currently, the compression flag(`connection.compression`) is not working due to the missing of `RequestAcceptEncoding` interceptor in HTTP client. ## Solution This PR added `RequestAcceptEncoding` interceptor in HTTP client builder...
## Problem While setting the document ID when indexing does provide exactly once delivery, it does put more load on Elasticsearch and is not necessary for all use cases. A...
