producer can not produce with KAFKA_STORAGE_ERROR

[mooc9988]

In long running test, producers can not send any record. logs of producers are:

08:25:48.490 [kafka-producer-network-thread | producer-135] WARN Sender - [Producer clientId=producer-135] Got error produce response with correlation id 5 on topic-partition testMain-2, retrying (2147483646 attempts left). Error: KAFKA_STORAGE_ERROR
08:25:48.490 [kafka-producer-network-thread | producer-135] WARN Sender - [Producer clientId=producer-135] Received invalid metadata error in produce request on partition testMain-2 due to org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on the disk.. Going to request metadata update now
08:25:48.490 [kafka-producer-network-thread | producer-135] WARN Sender - [Producer clientId=producer-135] Got error produce response with correlation id 6 on topic-partition testMain-2, retrying (2147483646 attempts left). Error: KAFKA_STORAGE_ERROR
08:25:48.490 [kafka-producer-network-thread | producer-135] WARN Sender - [Producer clientId=producer-135] Received invalid metadata error in produce request on partition testMain-2 due to org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on the disk.. Going to request metadata update now
08:25:48.490 [kafka-producer-network-thread | producer-135] WARN Sender - [Producer clientId=producer-135] Got error produce response with correlation id 7 on topic-partition testMain-2, retrying (2147483646 attempts left). Error: KAFKA_STORAGE_ERROR
08:25:48.490 [kafka-producer-network-thread | producer-135] WARN Sender - [Producer clientId=producer-135] Received invalid metadata error in produce request on partition testMain-2 due to org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on the disk.. Going to request metadata update now

inner producer in autobalancing also cannot produce:

[2023-12-09 09:05:09,204] WARN [Producer clientId=AutoBalancerMetricsReporterProducer] Got error produce response with correlation id 5638 on topic-partition __auto_balancer_metrics-0, retrying (0 attempts left). Error: KAFKA_STORAGE_ERROR (org.apache.kafka.clients.producer.internals.Sender)
[2023-12-09 09:05:09,204] WARN [Producer clientId=AutoBalancerMetricsReporterProducer] Received invalid metadata error in produce request on partition __auto_balancer_metrics-0 due to org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on the disk.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)

For servers, there are only STREAM_NOT_CLOSED exceptions.

Besides, inner consumers can not sync group:

server.log.2023-12-09-08:[2023-12-09 08:25:49,736] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer-7514695428000162748, groupId=AutoBalancerControllerConsumerGroup-group-7514695428000162748] Discovered group coordinator 10.0.1.149:9092 (id: 2147483644 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
server.log.2023-12-09-08:[2023-12-09 08:25:49,736] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer-7514695428000162748, groupId=AutoBalancerControllerConsumerGroup-group-7514695428000162748] Request joining group due to: rebalance failed due to 'This is not the correct coordinator.' (NotCoordinatorException) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
server.log.2023-12-09-08:[2023-12-09 08:25:49,736] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer-7514695428000162748, groupId=AutoBalancerControllerConsumerGroup-group-7514695428000162748] (Re-)joining group (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
server.log.2023-12-09-08:[2023-12-09 08:25:49,742] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer-7514695428000162748, groupId=AutoBalancerControllerConsumerGroup-group-7514695428000162748] Successfully joined group with generation Generation{generationId=71, memberId='AutoBalancerControllerConsumer-consumer-7514695428000162748-37abdb26-81ed-4498-b16e-14bc94dcdb51', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
server.log.2023-12-09-08:[2023-12-09 08:25:49,742] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer-7514695428000162748, groupId=AutoBalancerControllerConsumerGroup-group-7514695428000162748] Finished assignment for group at generation 71: {AutoBalancerControllerConsumer-consumer-7514695428000162748-37abdb26-81ed-4498-b16e-14bc94dcdb51=Assignment(partitions=[__auto_balancer_metrics-0])} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
server.log.2023-12-09-08:[2023-12-09 08:25:49,743] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer-7514695428000162748, groupId=AutoBalancerControllerConsumerGroup-group-7514695428000162748] SyncGroup failed: This is not the correct coordinator. Marking coordinator unknown. Sent generation was Generation{generationId=71, memberId='AutoBalancerControllerConsumer-consumer-7514695428000162748-37abdb26-81ed-4498-b16e-14bc94dcdb51', protocol='range'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
server.log.2023-12-09-08:[2023-12-09 08:25:49,743] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer-7514695428000162748, groupId=AutoBalancerControllerConsumerGroup-group-7514695428000162748] Group coordinator 10.0.1.149:9092 (id: 2147483644 rack: null) is unavailable or invalid due to cause: error response NOT_COORDINATOR. isDisconnected: false. Rediscovery will be attempted. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
server.log.2023-12-09-08:[2023-12-09 08:25:49,743] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer-7514695428000162748, groupId=AutoBalancerControllerConsumerGroup-group-7514695428000162748] Requesting disconnect from last known coordinator 10.0.1.149:9092 (id: 2147483644 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
server.log.2023-12-09-08:[2023-12-09 08:25:49,744] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer-7514695428000162748, groupId=AutoBalancerControllerConsumerGroup-group-7514695428000162748] Client requested disconnect from node 2147483644 (org.apache.kafka.clients.NetworkClient)
server.log.2023-12-09-08:[2023-12-09 08:25:49,745] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer-7514695428000162748, groupId=AutoBalancerControllerConsumerGroup-group-7514695428000162748] Discovered group coordinator 10.0.1.149:9092 (id: 2147483644 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
server.log.2023-12-09-08:[2023-12-09 08:25:49,745] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer-7514695428000162748, groupId=AutoBalancerControllerConsumerGroup-group-7514695428000162748] Group coordinator 10.0.1.149:9092 (id: 2147483644 rack: null) is unavailable or invalid due to cause: coordinator unavailable. isDisconnected: false. Rediscovery will be attempted. (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
server.log.2023-12-09-08:[2023-12-09 08:25:49,745] INFO [Consumer clientId=AutoBalancerControllerConsumer-consumer-7514695428000162748, groupId=AutoBalancerControllerConsumerGroup-group-7514695428000162748] Requesting disconnect from last known coordinator 10.0.1.149:9092 (id: 2147483644 rack: null) (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)

server.log.2023-12-09-08:[2023-12-09 08:25:49,741] INFO [GroupCoordinator 3]: Stabilized group AutoBalancerControllerConsumerGroup-group-7514695428000162748 generation 71 (__consumer_offsets-30) with 1 members (kafka.coordinator.group.GroupCoordinator)
server.log.2023-12-09-08:[2023-12-09 08:25:49,743] INFO [GroupCoordinator 3]: Assignment received from leader AutoBalancerControllerConsumer-consumer-7514695428000162748-37abdb26-81ed-4498-b16e-14bc94dcdb51 for group AutoBalancerControllerConsumerGroup-group-7514695428000162748 for generation 71. The group has 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
server.log.2023-12-09-08:[2023-12-09 08:25:49,743] INFO [GroupCoordinator 3]: Preparing to rebalance group AutoBalancerControllerConsumerGroup-group-7514695428000162748 in state PreparingRebalance with old generation 71 (__consumer_offsets-30) (reason: Error NOT_COORDINATOR when storing group assignment during SyncGroup (member: AutoBalancerControllerConsumer-consumer-7514695428000162748-37abdb26-81ed-4498-b16e-14bc94dcdb51)) (kafka.coordinator.group.GroupCoordinator)