Schema registry failing to start due to mismatch in metadata

[sauravkpaul]

we have 3 node cluster deployment with 3 kafka brokers and 2 schema-registry pods where the schema registry pods are failing to start after a upgrade and reboot. The error that we are getting is as below

{"log":"[2023-07-19 16:30:08,000] WARN [Producer clientId=producer-1] Received invalid metadata error in produce request on partition _schemas-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)\n","stream":"stdout","time":"2023-07-19T16:30:08.000435821Z"}
{"log":"[2023-07-19 16:30:08,004] ERROR Error starting the schema registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication)\n","stream":"stdout","time":"2023-07-19T16:30:08.010919567Z"}
{"log":"io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryInitializationException: Error initializing kafka store while initializing schema registry\n","stream":"stdout","time":"2023-07-19T16:30:08.010966003Z"}

As per what I read, this issue can be mitigated if the producer in schema registry does multiple retries then the metadata will be corrected automatically corrected and initialization may happen. Is there a config I can try to increase the producer retries. Any other suggestion to recover from this issue. Thanks

[maixiaohai]

seems retry configuration was removed by this commit 63c59306d2dfbe349717cfe697e7f5831c74d553