Kafka connector doesn't move data to Elasticsearch index

[pollynesterovich]

Hey, We are using an kafka connector elasticsearch, with the following configuration:

connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector type.name=_doc transforms.insertRecordTimestamp.timestamp.field=@timestamp tasks.max=2 transforms.convertRecordTimestamp.target.type=string transforms.convertRecordTimestamp.type=org.apache.kafka.connect.transforms.TimestampConverter$Value connection.timeout.ms=50000 transforms=routeTS,insertRecordTimestamp,convertRecordTimestamp transforms.routeTS.topic.format=<name>-${timestamp}-s max.retries=1000 key.ignore=true retry.backoff.ms=30000 max.buffered.records=100000 read.timeout.ms=10000 topics=<name> transforms.routeTS.type=org.apache.kafka.connect.transforms.TimestampRouter batch.size=10000 transforms.convertRecordTimestamp.format=yyyy-MM-dd'T'HH:mm:ss.SSSZ max.in.flight.requests=10 transforms.convertRecordTimestamp.field=@timestamp schema.ignore=true transforms.routeTS.timestamp.format=yyyyMMdd flush.timeout.ms=90000 transforms.insertRecordTimestamp.type=org.apache.kafka.connect.transforms.InsertField$Value topic.index.map=<name_index> connection.url=<http://ip:9200> linger.ms=50

From time to time we run into a problem where don't see documents in elasticsearch index. The state of the kafka connector is running. After restarting the task, documents begin to gradually load into elasticsearch. Maybe someone has suggestions for this behavior and solutions to the problem?

[bbondarets]

we faced with similiar issue. only first 8K messages are flushed into elastic, but others are not. Do you have any updates on this?

[tporeba]

I experience similar issues on ElasticsearchSinkConnector, state is "running" but connector task seems to hang doing nothing. Eventually kafka is kicking it out of consumer group. Maybe broken retries are the cause https://github.com/confluentinc/kafka-connect-elasticsearch/pull/575 ?