Elasticsearch sink connector showing high CPU usage

[kushaqflier]

I am using elastic search sink connector in distributed mode(2 instance). With task of 8 and about 20 to 25 topics to be sink’ed to elastic search.

Even when there are no records to sink, the worker java process is showing 100% CPU usage using xmx(4GB).

The end to end transfer of records is happening properly, but high CPU usage is a concern.

Tried with various time related consumer settings (batch size, max poll interval) even enabled error tolerance with dead letter queue, none of them helped.

my connector settings:

 {
    “name”: “elasticsearch-sink”,
    "config”: {
    “connector.class”: “io.confluent.connect.elasticsearch.ElasticsearchSinkConnector”,
    “tasks.max”: “8”,
    “topics.regex”:"(mytopics_\d+$)",
    “key.ignore”: “true”,
    “schema.ignore”: “true”,
    “connection.url”: “http://eshost:esport”,
    “type.name”: “kafka-connect”
   }
}

worker settings:

bootstrap.servers=localhost:9094,localhost:9095
group.id=test-cluster
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
offset.storage.topic=connect-offsets
offset.storage.replication.factor=3
config.storage.topic=connect-configs
config.storage.replication.factor=3
status.storage.topic=connect-status
status.storage.replication.factor=3
status.storage.partitions=8
rest.port=9034
plugin.path=/pluginpath
log4j.rootLogger=DEBUG, stdout

I’m trying it on a server grade setup (64GB RAM and 8 vCPU cores) and there is good connectivity to Kafka Broker as well as Elasticsearch server host.

I tried to isolate the issue by using FileStream sink connector instead of Elasticsearch connector and this issue goes off.

[Abdulsametileri]

Any update?