Enable topic configs (like retention.ms) to be configurable for internal topics.

[NK91]

When trying to connect KSQL server (confluentinc/cp-ksql-server:5.3.2) to my IBM Cloud-based Kafka Cluster (Event Stream), I received the error:

[2020-01-02 14:32:45,120] INFO Creating topic '_confluent-ksql-default__command_topic' (io.confluent.ksql.services.KafkaTopicClient:106)
[2020-01-02 14:32:45,290] INFO Server shutting down (io.confluent.ksql.rest.server.KsqlServerMain:78)
[2020-01-02 14:33:00,320] INFO Stopped NetworkTrafficServerConnector@33a630fa{HTTP/1.1,[http/1.1]}{0.0.0.0:8088} (org.eclipse.jetty.server.AbstractConnector:341)
[2020-01-02 14:33:00,321] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:167)
[2020-01-02 14:33:00,324] INFO Stopped o.e.j.s.ServletContextHandler@29f0802c{/ws,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:1045)
[2020-01-02 14:33:00,331] INFO Stopped o.e.j.s.ServletContextHandler@3a60c416{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:1045)
[2020-01-02 14:33:00,334] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:62)
io.confluent.ksql.exception.KafkaResponseGetFailedException: Failed to guarantee existence of topic _confluent-ksql-default__command_topic
	at io.confluent.ksql.services.KafkaTopicClientImpl.createTopic(KafkaTopicClientImpl.java:122)
	at io.confluent.ksql.rest.util.KsqlInternalTopicUtils.ensureTopic(KsqlInternalTopicUtils.java:108)
	at io.confluent.ksql.rest.server.KsqlRestApplication.initialize(KsqlRestApplication.java:262)
	at io.confluent.ksql.rest.server.KsqlRestApplication.startKsql(KsqlRestApplication.java:223)
	at io.confluent.ksql.rest.server.KsqlRestApplication.start(KsqlRestApplication.java:210)
	at io.confluent.ksql.rest.server.KsqlServerMain.tryStartApp(KsqlServerMain.java:74)
	at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:60)
Caused by: org.apache.kafka.common.errors.PolicyViolationException: Invalid retention.ms specified. The allowed range is [3600000..2592000000]

Is it possible to change this retention config for internal topic?

[apurvam]

This isn't possible to configure at this time. Currently you can only configure the replication factor and min.isr for internal topics (see: https://github.com/confluentinc/ksql/blob/master/ksql-common/src/main/java/io/confluent/ksql/util/KsqlConfig.java).

Marking this as a feature request.

[JaspalKaurBatra]

@NK91 @apurvam Hi, I am facing the same issue. Is there any fix for this? I am stuck not sure how to proceed further, Is there any alternate approach?

[AngelVilardell]

Hello,

I was facing the same issue. Internal topics grew up like TB of data in a matter of days with infinite retention by default. We altered them setting retention.ms to some value instead of infinite (-1) but after that everything broke. Today I executed this command: set topic.retention.ms=3600000 After that, I created a table and all related internal topics were created with retention.ms=1h instead of infinite. Will try next week in prd environment to see if ksqldb (0.28.2) evicts segments and everything is ok. Source: https://docs.confluent.io/platform/current/streams/developer-guide/config-streams.html#internal-topic-parameters Hope it helps Regards

[itsmestefanjay]

@apurvam are there any news on this?

[ubiquitousbyte]

This feature is really needed.