remora icon indicating copy to clipboard operation
remora copied to clipboard

Remora not sending metrics if one of the endpoints down

Open arnoldyahad opened this issue 5 years ago • 0 comments

we have multiple endpoints to our kafka cluster

version: "2"
services:

  remora-kafka:
    image: remora:latest
    environment:
      KAFKA_ENDPOINT: "kafka1.our_domain:9092,kafka2.our_domain:9092,kafka3.our_domain:9092,kafka4.our_domain:9092"
      DATADOG_ON: "true"
      SERVER_PORT: "9019"
      TO_REGISTRY: "true"
      DATADOG_AGENT_HOST: "localhost"
      DATADOG_REMOVE_TAGS_FROM_METRIC_NAME: "true"
    ports:
      - 9019:9019
    restart: always
    network_mode: "host"

when one of the endpoint brokers failed ( shut down because of any reason ) instead of working against one of the other brokers it just starting to send only its failure in the logs

2019-05-20 08:31:49,513 - [DEBUG] - [remora-kafka-consumer-dispatcher-83] kafka.admin.AdminClient - Request FIND_COORDINATOR failed against node kafka2.our_domain:9092 (id: -4 rack: null)
org.apache.kafka.common.errors.DisconnectException: null
2019-05-20 08:31:49,515 - [DEBUG] - [remora-kafka-consumer-dispatcher-85] kafka.admin.AdminClient - Request FIND_COORDINATOR failed against node kafka2.our_domain:9092 (id: -5 rack: null)
org.apache.kafka.common.errors.DisconnectException: null
2019-05-20 08:31:49,515 - [DEBUG] - [remora-kafka-consumer-dispatcher-85] kafka.admin.AdminClient - Request FIND_COORDINATOR failed against node kafka2.our_domain:9092 (id: -4 rack: null)

any way to make it work with the other working brokers?

thanks a lot, Arnold

arnoldyahad avatar May 20 '19 09:05 arnoldyahad