[FLINK-33045] Make it possible to disable auto-registering schema in Schema Registry

[MartijnVisser]

What is the purpose of the change

This PR is based on https://github.com/apache/flink/pull/25410 and aims to complete the necessary tasks. It introduces auto.register.schemas as a table option. Compared to the linked PR, it includes unit tests, a new IT case and updated documentation

Brief change log

• Introduces new table option auto.register.schemas
• Adds unit tests
• Adds a new AvroConfluentITCase
• Removes previous (currently disabled) bash-based tests
• It also bumps certain dependencies

Verifying this change

This change added tests and can be verified as follows:

• Run AvroConfluentITCase that writes and reads from/to Kafka using avro-confluent, with the table option set to true (default) to show that Flink can register the schema and false where it relies on schema registration outside of Flink

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): yes
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? docs / JavaDocs

[flinkbot]

CI report:

• c2d944a385d1f0889888da6737c9d1ec924d326c Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[MartijnVisser]

@flinkbot run azure

[MartijnVisser]

@flinkbot run azure

[raphaelauv]

hi , will this PR correct this problem ? -> https://stackoverflow.com/questions/79738462/flink-confluentregistryavroserializationschema-not-respecting-registryconfigs ? thanks