kafka-connect-elasticsearch
kafka-connect-elasticsearch copied to clipboard
confluentinc
Kafka Connect Elasticsearch connector
## Problem Currently as per limitations its not possible to use `RegexRouter` to specify the target index name without compromising on performance ( asynchronous writes must be disabled ). ##...
With a configuration such as ```json "max.retries": "10", "retry.backoff.ms": "500" ``` I would expect the connector to continue retrying up to 6-8 minutes given the number of retries configured and...
When `TOO_MANY_REQUESTS` happen, the connector does not seem to log this as the error is bubbled up to Kafka connect instead It would be useful if we could log that...
## Problem Keeping the dependencies up to date is crucial to ensure that the project stays secure and updated. ## Solution See https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/ ##### Does this solution apply anywhere else?...
I am running in a shared environment where the abstraction in terms of `tasks`, `batch.size`, etc is not enough to control the volume in my system. My system enforces a...
Deployed a Kafka cluster using strimzi on the cloud. Used this KafkaConnect config to get the elasticsearch plugin files. ``` apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaConnect metadata: name: es-connect-cluster annotations: strimzi.io/use-connector-resources: "true"...
When using the default `AsyncOffsetTracker` (`flush.synchronously=false`), if the topic contains a lot of null values, the consumer get paused indefinitely. Logs: ``` [2023-10-10 13:20:52,259] TRACE Ignoring record from topic=real-time-analytics.package-tree.state partition=5...
## Problem When UPSERT mode DELETE still sends the version s external that is causing issue. ELASTIC index snapshot and restore to different environment where Kafka offset values are not...
## Problem ## Solution ##### Does this solution apply anywhere else? - [ ] yes - [ ] no ##### If yes, where? ## Test Strategy ##### Testing done: -...
Hi, We are attempting to sink a topic into an elasticsearch cluster, and we want tombstone records to delete docs out of the cluster. We are able to sink all...
