KafkaIndexSupervisor becomes UNHEALTHY, blocking partition consumption

[599166320]

Druid Version

24.0.1

Supervisor Type

KafkaIndexSupervisor

Problem Description

We observed that when a Kafka ingestion task's duration exceeds the configured segmentGranularity, the Supervisor enters the UNHEALTHY_SUPERVISOR state. As a result, certain Kafka partitions stop being consumed, even if the partition contains a very small amount of data (only a few thousand rows).

This occurs even under minimal load, which indicates that the issue is likely related to Supervisor's internal task handling rather than Kafka volume or resource exhaustion.

Additionally, the logs sometimes show repeated exceptions like: java.lang.IllegalStateException: Optional.get() cannot be called on an absent value