flink-connector-aws icon indicating copy to clipboard operation
flink-connector-aws copied to clipboard

[FLINK-24379][Formats] Avro Glue Schema Registry table format

Open nicusX opened this issue 1 year ago • 3 comments

Purpose of the change

Implement Table API support for Avro with AWS Glue Schema Registry

Verifying this change

  • Added unit tests
  • Manually verified by running a Flink application locally against a local Kafka cluster and using GSR.

Significant changes

  • [x] Dependencies have been added or upgraded
  • [ ] Public API has been changed (Public API is any class annotated with @Public(Evolving))
  • [ ] Serializers have been changed
  • [X] New feature has been introduced
    • If yes, how is this documented? docs

The implementation is mostly based on the original PR to the main Flink repo with minor changes. SQL uber-jar has been added and docs fixed and enriched.

nicusX avatar Dec 29 '23 11:12 nicusX

Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)

boring-cyborg[bot] avatar Dec 29 '23 11:12 boring-cyborg[bot]

@dannycranmer please review this PR when you have the chance

nicusX avatar Jan 04 '24 14:01 nicusX

@z3d1k I addressed most of the comments. But I am not sure how to implement the IT, in particular for a format using an external registry. Any reference or example I can follow? Most of formats lack of IT. And this one requires an external system

The only similar format, flink-avro-confluent-registry, doesn't have any IT.

The GRS Serialization/DeserializationSchema IT relies on a AWS account that I assume is available during the CI/CD build. Any reference on how to leverage this?

nicusX avatar May 13 '24 07:05 nicusX