ksql fix: ensure queries using JSON

Description

Addresses #7707. With the JsonSchema format, upgradeability is determined by the additionalProperties flag on the schema. Somewhat counterintuitively, this value must be false in order to support backward compatibility when new properties are added to an existing schema.

Prior to https://github.com/confluentinc/schema-registry/pull/1968, there was no way to configure the ConnectSchema-to-JsonSchema conversion such as to set the additionalProperties value. Because the default value of the property is true, the effect has been that all JsonSchema schemas created by ksqlDB are non-evolvable, meaning query upgrades are not possible when JSON_SR is used. See #7707 for the symptoms.

This PR configures the JsonSchema converter so that object schemas are registered with the additionalProperties flag set to false. In cases (presumably very unlikely) where users wish to prevent the schema from being evolved (and thus the query from being upgraded), the schema can be later updated to toggle the additionalProperties flag via the Schema Registry REST API or Confluent UI.

Testing done

Added unit test.

Reviewer checklist

[ ] Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
[ ] Ensure relevant issues are linked (description should include text like "Fixes #")

Jul 26 '21 20:07 colinhicks

Test failures are known flaky tests:

io.confluent.ksql.cli.CliTest.testPersistentSelectStar
io.confluent.ksql.cli.CliTest.shouldRunScriptOnRunScript
io.confluent.ksql.cli.CliTest.shouldRunScriptOnRunInteractively
io.confluent.ksql.cli.CliTest.shouldHandlePullQuery

Jul 29 '21 13:07 colinhicks

I was looking into the feasibility of adding a YATT-based test in order to assert that we can insert values after updating an existing JSON_SR-based stream with a new column. It looks like this won't be possible without first addressing https://github.com/confluentinc/ksql/issues/6059.

Jul 29 '21 14:07 colinhicks

Any movement on this?

Sep 15 '22 13:09 maxbol

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}