KSQL - Table is not being created properly for Compact topic

[karthikeyanrd27]

Please consider the following example and help us to resolve this is issue. We have 2 topic . One topic is transactional topic and another one is compact topic . Both are in AVRO Format. . Just replicating the transactional topic as compact topic . Message content and format all are same except Compact nature.

1. Transactional Topic : TBL_MS_TOPIC

bin/kafka-topics  --describe --zookeeper localhost:2181 --topic TBL_MS_TOPIC
Topic:TBL_MS_TOPIC	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: TBL_MS_TOPIC	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

2. Compact topic : TBL_MS_TOPIC_COMPACT

bin/kafka-topics  --describe --zookeeper localhost:2181 --topic TBL_MS_TOPIC_COMPACT
Topic:TBL_MS_TOPIC_COMPACT1	PartitionCount:1	ReplicationFactor:1	Configs:cleanup.policy=compact
	Topic: NBC_APPS.TBL_MS_TOPIC_COMPACT1	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

Trying to create KSQL table for transactional topic TBL_MS_TOPIC.

ksql> CREATE STREAM TBL_MS_TOPIC_RAW_STREAM  WITH (KAFKA_TOPIC='TBL_MS_TOPIC', VALUE_FORMAT='AVRO');
Message        
----------------
 Stream created 
----------------

ksql> CREATE STREAM TBL_MS_TOPIC_RAW_STREAM_REKEY  AS SELECT * FROM TBL_MS_TOPIC_RAW_STREAM PARTITION BY PROPERTY_ID;

 Message                    
----------------------------
 Stream created and running 
----------------------------

ksql> CREATE TABLE TBL_MS_TOPIC  WITH (KAFKA_TOPIC='TBL_MS_TOPIC_RAW_STREAM_REKEY', VALUE_FORMAT='avro', KEY='PROPERTY_ID');

 Message       
---------------
 Table created 
---------------

ksql> select * from TBL_MS_TOPIC;
1537806207755 | 10009 | TBL_MS_TOPIC | D | 2018-07-18 18:51:34.000000 | 2018-07-18 14:51:39.397000 | 00000003470236836100 | 10009 | Bravo | 1806 | -1 | 3202000 | NULL | Bravo | 1991 | 30-AUG-18 | 3111001

Able to create successfully without any issue for transactional topic TBL_MS_TOPIC.When we are trying to create table for Compact topic TBL_MS_TOPIC_COMPACT is not working. TBL_MS_TOPIC_RAW_STREAM_REKEY with partition is not working for compact nature Topic.

ksql> CREATE STREAM TBL_MS_TOPIC_RAW_STREAM WITH (KAFKA_TOPIC='TBL_MS_TOPIC_COMPACT', VALUE_FORMAT='AVRO');

 Message        
----------------
 Stream created 
----------------

ksql> CREATE STREAM TBL_MS_TOPIC_RAW_STREAM_REKEY  AS SELECT * FROM TBL_MS_TOPIC_RAW_STREAM PARTITION BY PROPERTY_ID;

 Message                    
----------------------------
 Stream created and running 
----------------------------

ksql> set 'auto.offset.reset' = 'earliest';
Successfully changed local property 'auto.offset.reset' from 'null' to 'earliest'

`ksql> print 'TBL_MS_TOPIC_RAW_STREAM_REKEY' from beginning;


`

ksql> CREATE TABLE TBL_MS_TOPIC WITH (KAFKA_TOPIC='TBL_MS_TOPIC_RAW_STREAM_REKEY', VALUE_FORMAT='avro', KEY='PROPERTY_ID');
 Unable to verify the AVRO schema is compatible with KSQL. Subject not found.; error code: 40401 

Thanks & Regards

Karthikeyan Rasipalayam Durairaj

[vcrfxia]

Hi @karthikeyanrd27 -- were you able to resolve this issue? If not, a question for you in order to help triage: is there new data being produced to the topic TBL_MS_TOPIC_COMPACT? I see that you ran set 'auto.offset.reset' = 'earliest'; after creating the stream TBL_MS_TOPIC_RAW_STREAM_REKEY, so if there is no new data being produced to TBL_MS_TOPIC_COMPACT, that would explain why the stream TBL_MS_TOPIC_RAW_STREAM_REKEY is empty.

The fact that TBL_MS_TOPIC_RAW_STREAM_REKEY is empty explains why the final CREATE TABLE statement failed: KSQL creates schemas in Schema Registry only when the first record is written to a particular topic, so if no such record was written, then no schema is created in Schema Registry and thus the CREATE TABLE command hits a 404.

[RashidAther]

Hi, Can you share the resolution for the above. I was following this document: https://docs.confluent.io/legacy/platform/5.1.4/ksql/docs/developer-guide/syntax-reference.html#ksql-key-requirements