[kinesis][glue] Updating AWS SDK dependencies to latest versions

[dannycranmer]

What is the purpose of the change

• Updating AWS SDK dependencies to latest versions to pull recent bug fixes and add support for new AWS regions.
• Removing the dependency on AWS Regions enum (for AWS SDK v1) so that region support for new AWS regions is automatic, and wouldn't need a PR.

Brief change log

This is a backport of https://github.com/apache/flink/pull/20476 and https://github.com/apache/flink/pull/20496 from 1.16 to 1.13.

Upgrading dependencies:

• AWS SDK v2 from 2.16.86 to 2.17.247
• AWS SDK v1 from 1.12.7 to 1.12.276
• Kinesis Consumer Library from 1.11.1 to 1.14.8
• Kinesis Producer Library from 0.14.0 to 0.14.1

For the following connectors/formats:

• Kinesis Data Streams
• Kinesis Data Firehose
• Glue Schema Registry AVRO

• Change calls from Regions class to calls from RegionUtils class

Verifying this change

• Unit tests/e2e tests
• Manual regression
  • flink-glue-schema-registry-test :: mvn clean test -Pe2e-travis1 (providing GSR keys)
  • flink-connector-kinesis :: Verified with local test app (EFO source/Polling source/Sink)
• Manually verified that Kinesis connector with this fix works in region not present in Regions enum

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): yes
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• 5f511f594ff54435211d4bcdf85b57443feb5d92 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[dannycranmer]

@flinkbot run azure

[dannycranmer]

@flinkbot run azure

[dannycranmer]

@flinkbot run azure

[zentol]

Why are you trying to update 1.13? That version is no longer supported.

[zentol]

The 1.13 branch is missing https://github.com/apache/flink/pull/20044; that's why it is failing.

[dannycranmer]

Why are you trying to update 1.13? That version is no longer supported.

Ack, I could not find the policy, but have found the docs:

• https://flink.apache.org/downloads.html#update-policy-for-old-releases

The reason for the change is that Amazon Kinesis Data Analytics (KDA) has recently launched in the Jakarta region. KDA still supports Flink 1.13 and without this change there is no Kinesis Data Streams connector available in Maven central that works in this region. Unfortunately the AWS SDK is shaded into the connector and therefore cannot (easily) be overridden from the user application.

I will close this PR and work out a way forward.