schema-registry
schema-registry copied to clipboard
confluentinc
Got "ERROR The retention policy of the schema topic _schemas is incorrect"?
Hi,
I have 4 machines among this 3 brokers have installed both zookeeper and kafka and in remain one machine installed kafka-rest and schema-registry. Below is the schema registry properties.
listeners=http://0.0.0.0:8081 kafkastore.connection.url=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181/kafka kafkastore.topic=_schemas debug=false
But when i try to start schema registry
$CONFLUENT_HOME/bin/schema-registry-start $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties i am getting the following error.
ERROR The retention policy of the schema topic _schemas is incorrect. You must configure the topic to 'compact' cleanup policy to avoid Kafka deleting your schemas after a week. Refer to Kafka documentation for more details on cleanup policies (io.confluent.kafka.schemaregistry.storage.KafkaStore:388) [2017-12-28 05:04:48,102] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:52) java.lang.IllegalStateException: The retention policy of the schema topic _schemas is incorrect. Expected cleanup.policy to be 'compact' but it is null at io.confluent.kafka.schemaregistry.storage.KafkaStore.verifySchemaTopic(KafkaStore.java:393) at io.confluent.kafka.schemaregistry.storage.KafkaStore.createSchemaTopic(KafkaStore.java:283) at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:152) at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:212) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:56) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.setupResources(SchemaRegistryRestApplication.java:38) at io.confluent.rest.Application.createServer(Application.java:157) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:44)
Similar error (docker image v4.0.0)
[2018-01-11 14:37:49,682] INFO Validating schemas topic _schemas (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2018-01-11 14:37:49,773] ERROR The retention policy of the schema topic _schemas is incorrect. You must configure the topic to 'compact' cleanup policy to avoid Kafka deleting your schemas after a week. Refer to Kafka documentation for more details on cleanup policies (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2018-01-11 14:37:49,780] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry)
[2018-01-11 14:37:49,783] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain)
java.lang.NullPointerException
at io.confluent.kafka.schemaregistry.storage.KafkaStore.close(KafkaStore.java:366)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.close(KafkaSchemaRegistry.java:720)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.onShutdown(SchemaRegistryRestApplication.java:111)
I'm thinking it only happens sometimes and perhaps after restarting kafka cluster or when schema-registry is restarting often.
Manually fixed with kafka-topics --alter --config cleanup.policy=compact --topic _schemas --zookeeper dev-zookeeper/kafka
@lwu08 One reason that might cause this problem is when you accidentally delete _schemas topic.
kafka-topics is deprecated
You should use
kafka-configs.sh --zookeeper localhost --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compact
Still only workaround
below command worked for me .. thanks bud ... kafka-configs.sh --zookeeper localhost --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compact
We are using helm to deploy kafka,zookeeper and schema registry at the same time
The majority of time, the initial startup attempt of schema-registry throws the following error
[2019-12-19 02:22:56,304] INFO Kafka version: 5.3.1-ce (org.apache.kafka.common.utils.AppInfoParser)
[2019-12-19 02:22:56,304] INFO Kafka commitId: 30df4bec2f6882ab (org.apache.kafka.common.utils.AppInfoParser)
[2019-12-19 02:22:56,304] INFO Kafka startTimeMs: 1576722176301 (org.apache.kafka.common.utils.AppInfoParser)
[2019-12-19 02:23:00,352] INFO Creating schemas topic _schemas (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2019-12-19 02:23:01,849] WARN Creating the schema topic _schemas using a replication factor of 1, which is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic. (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2019-12-19 02:23:03,554] INFO Validating schemas topic _schemas (io.confluent.kafka.schemaregistry.storage.KafkaStore)
[2019-12-19 02:23:04,561] ERROR Error starting the schema registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication)
io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryInitializationException: Error initializing kafka store while initializing schema registry
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:224)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.initSchemaRegistry(SchemaRegistryRestApplication.java:66)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.configureBaseApplication(SchemaRegistryRestApplication.java:81)
at io.confluent.rest.Application.createServer(Application.java:205)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:42)
Caused by: io.confluent.kafka.schemaregistry.storage.exceptions.StoreInitializationException: Failed trying to create or validate schema topic configuration
at io.confluent.kafka.schemaregistry.storage.KafkaStore.createOrVerifySchemaTopic(KafkaStore.java:179)
at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:116)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:222)
... 4 more
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
at io.confluent.kafka.schemaregistry.storage.KafkaStore.verifySchemaTopic(KafkaStore.java:237)
at io.confluent.kafka.schemaregistry.storage.KafkaStore.createSchemaTopic(KafkaStore.java:222)
at io.confluent.kafka.schemaregistry.storage.KafkaStore.createOrVerifySchemaTopic(KafkaStore.java:171)
... 6 more
Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.```
Subsequent restarts of schema reg, throws the error in the first comment
And there again, sometimes it works fine.
We are using helm to deploy kafka,zookeeper and schema registry at the same time
The majority of time, the initial startup attempt of schema-registry throws the following error
[2019-12-19 02:22:56,304] INFO Kafka version: 5.3.1-ce (org.apache.kafka.common.utils.AppInfoParser) [2019-12-19 02:22:56,304] INFO Kafka commitId: 30df4bec2f6882ab (org.apache.kafka.common.utils.AppInfoParser) [2019-12-19 02:22:56,304] INFO Kafka startTimeMs: 1576722176301 (org.apache.kafka.common.utils.AppInfoParser) [2019-12-19 02:23:00,352] INFO Creating schemas topic _schemas (io.confluent.kafka.schemaregistry.storage.KafkaStore) [2019-12-19 02:23:01,849] WARN Creating the schema topic _schemas using a replication factor of 1, which is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic. (io.confluent.kafka.schemaregistry.storage.KafkaStore) [2019-12-19 02:23:03,554] INFO Validating schemas topic _schemas (io.confluent.kafka.schemaregistry.storage.KafkaStore) [2019-12-19 02:23:04,561] ERROR Error starting the schema registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication) io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryInitializationException: Error initializing kafka store while initializing schema registry at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:224) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.initSchemaRegistry(SchemaRegistryRestApplication.java:66) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.configureBaseApplication(SchemaRegistryRestApplication.java:81) at io.confluent.rest.Application.createServer(Application.java:205) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:42) Caused by: io.confluent.kafka.schemaregistry.storage.exceptions.StoreInitializationException: Failed trying to create or validate schema topic configuration at io.confluent.kafka.schemaregistry.storage.KafkaStore.createOrVerifySchemaTopic(KafkaStore.java:179) at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:116) at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:222) ... 4 more Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition. at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272) at io.confluent.kafka.schemaregistry.storage.KafkaStore.verifySchemaTopic(KafkaStore.java:237) at io.confluent.kafka.schemaregistry.storage.KafkaStore.createSchemaTopic(KafkaStore.java:222) at io.confluent.kafka.schemaregistry.storage.KafkaStore.createOrVerifySchemaTopic(KafkaStore.java:171) ... 6 more Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.``` Subsequent restarts of schema reg, throws the error in the first comment And there again, sometimes it works fine.
Ditto, this is how we deploy too, and this is pain in the *** , everytime registry crashes or reboots we end up in this situation, and how come this issues still a thing even after 3 years. Really sad state of this project.
Is there a solution for this? It happens when schema registry restarts
Is there a permanent solution for this? It is happening on every schema registry restart
kafka-topicsis deprecated You should usekafka-configs.sh --zookeeper localhost --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compactStill only workaround
Thanks for the information, just would like to add solution for kafka in kubernetes:
kubectl -n namespace exec kafka-pod -- bin/kafka-configs.sh --alter --entity-type topics --entity-name _schemas --add-config cleanup.policy=compact --bootstrap-server localhost:9092
kafka-configs.sh --zookeeper localhost --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compact
Awesome, this works.
I was not able to find kafka-configs.sh and "kafka-configs" present in CP 7.2.2 didn't help. I had to manually delete the topic and then create one with the changed cleanup policy.
kafka-topics --bootstrap-server=127.0.0.1:9092 --delete --topic _schemas
kafka-topics --bootstrap-server=127.0.0.1:9092 --create --topic _schemas --partitions=1 --replication-factor=3 --config cleanup.policy=compact
Its been 6 years and is still an issue. Can we get this fixed ASAP please?
