POST /topics/:topic returns 200 on error

[TrueWill]

Compacted topics require keys when producing. If you forget to specify a key, the Kafka CLI tool kafka-console-producer returns a helpful error within a few seconds:

CorruptRecordException: This message has failed its CRC checksum, exceeds the valid size, has a null key for a compacted topic, or is otherwise corrupt.

If you forget to specify a key with POST /topics/(string:topic_name), though, the call takes 2 minutes (120000 ms) and returns 200 OK.

The only way to detect this is to check error_code and error in the response.

At a minimum I don't think this should return 200 (maybe 422 Unprocessable Entity?). This should also be noted in the documentation under Status Codes. It would be nice if it detected the error and returned in a shorter period of time, but that's a minor concern.

To replicate:

(Using cp-all-in-one)

#!/bin/sh
host="http://localhost:8082"

echo 'Pre-created COMPACT topic with 1 partition and 1 replica.'

date +%M%S
echo "\nProducing a message WITHOUT the required key..."
curl -i -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" \
    --data '{"records":[{"value":"Value X"}]}' \
    "$host/topics/topic-key-error"

echo "\n\nDone."
date +%M%S

Output

Pre-created COMPACT topic with 1 partition and 1 replica.
2134

Producing a message WITHOUT the required key...
HTTP/1.1 200 OK
Date: Wed, 11 Sep 2019 14:21:34 GMT
Content-Type: application/vnd.kafka.v1+json
Vary: Accept-Encoding, User-Agent
Content-Length: 204
Server: Jetty(9.4.18.v20190429)

{"offsets":[{"partition":null,"offset":null,"error_code":50003,"error":"Expiring 1 record(s) for topic-key-error-0:120000 ms has passed since batch creation"}],"key_schema_id":null,"value_schema_id":null}

Done.
2334

[TrueWill]

A coworker wondered what would happen if we sent two records, one good and one bad:

--data '{"records":[{"key":"3984983","value":"Value 3984983"},{"value":"Value BAD"}]}'

HTTP/1.1 200 OK

{"offsets":[{"partition":null,"offset":null,"error_code":50003,"error":"Expiring 2 record(s) for topic-key-error-0:120001 ms has passed since batch creation"},{"partition":null,"offset":null,"error_code":50003,"error":"Expiring 2 record(s) for topic-key-error-0:120001 ms has passed since batch creation"}],"key_schema_id":null,"value_schema_id":null}

[TrueWill]

Even with multiple partitions and multiple records, it appears to be all-or-nothing:

--data '{"records":[{"key":"B","value":"Value B2"},{"value":"Value BAD"}]}'

HTTP/1.1 200 OK

{"offsets":[{"partition":null,"offset":null,"error_code":50003,"error":"Expiring 2 record(s) for compact-multi-partition-0:120001 ms has passed since batch creation"},{"partition":null,"offset":null,"error_code":50003,"error":"Expiring 2 record(s) for compact-multi-partition-0:120001 ms has passed since batch creation"}],"key_schema_id":null,"value_schema_id":null}

[MarkC0x]

Hi @TrueWill

I tried to reproduce this on CP 5.5.0 and it looks to have been fixed. If I attempt to send a record without a key to a compacted topic it now returns right away with an InvalidRecordException and also displays the following message:

This record has failed the validation on broker and hence be rejected.

Please let us know if you are still seeing the previously described behavior on your side?

[TrueWill]

@MarkC0x Thank you - we're in the process of moving off of the REST Proxy and we're no longer using compacted topics with it, so I no longer have a straightforward way of verifying this.