Bump apache-beam[gcp] from 2.28.0 to 2.40.0 in /megalist_dataflow

[dependabot[bot]]

trafficstars

Bumps apache-beam[gcp] from 2.28.0 to 2.40.0.

Release notes

Sourced from apache-beam[gcp]'s releases.

v2.40.0

We are happy to present the new 2.40.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.40.0 check out the detailed release notes.

Highlights

• Added RunInference API, a framework agnostic transform for inference. With this release, PyTorch and Scikit-learn are supported by the transform. See also example at apache_beam/examples/inference/pytorch_image_classification.py

I/Os

• Upgraded to Hive 3.1.3 for HCatalogIO. Users can still provide their own version of Hive. (Java) (Issue-19554).

New Features / Improvements

• Go SDK users can now use generic registration functions to optimize their DoFn execution. (BEAM-14347)
• Go SDK users may now write self-checkpointing Splittable DoFns to read from streaming sources. (BEAM-11104)
• Go SDK textio Reads have been moved to Splittable DoFns exclusively. (BEAM-14489)
• Pipeline drain support added for Go SDK has now been tested. (BEAM-11106)
• Go SDK users can now see heap usage, sideinput cache stats, and active process bundle stats in Worker Status. (BEAM-13829)
• The serialization (pickling) library for Python is dill==0.3.1.1 (BEAM-11167)

Breaking Changes

• The Go Sdk now requires a minimum version of 1.18 in order to support generics (BEAM-14347).
• synthetic.SourceConfig field types have changed to int64 from int for better compatibility with Flink's use of Logical types in Schemas (Go) (BEAM-14173)
• Default coder updated to compress sources used with BoundedSourceAsSDFWrapperFn and UnboundedSourceAsSDFWrapper.

Bugfixes

• Fixed X (Java/Python) (BEAM-X).
• Fixed Java expansion service to allow specific files to stage (BEAM-14160).
• Fixed Elasticsearch connection when using both ssl and username/password (Java) (BEAM-14000)

Detailed list of PRs

• [BEAM-14048] [CdapIO] Add ConfigWrapper for building CDAP PluginConfigs by @​Amar3tto in apache/beam#17051
• [BEAM-14196] add test verifying output watermark propagation in bundle by @​je-ik in apache/beam#17504
• Move master readme.md to 2.40.0 by @​y1chi in apache/beam#17552
• [BEAM-14173] Fix Go Loadtests on Dataflow & partial fix for Flink by @​lostluck in apache/beam#17554
• Upgrade python sdk container requirements. by @​y1chi in apache/beam#17549
• [BEAM-11205] Update Libraries BOM dependencies to version 25.2.0 by @​benWize in apache/beam#17497
• [BEAM-12603] Add retry on grpc data channel and remove retry from test. by @​y1chi in apache/beam#17537
• [BEAM-14303] Add a way to exclude output timestamp watermark holds by @​reuvenlax in apache/beam#17359
• [BEAM-14347] Allow users to optimize DoFn execution with a single generic registration function by @​damccorm in apache/beam#17429
• [BEAM-5878] Add (failing) kwonly-argument test by @​TheNeuralBit in apache/beam#17509

... (truncated)

Changelog

Sourced from apache-beam[gcp]'s changelog.

[2.40.0] - 2022-06-25

Highlights

• Added RunInference API, a framework agnostic transform for inference. With this release, PyTorch and Scikit-learn are supported by the transform. See also example at apache_beam/examples/inference/pytorch_image_classification.py

I/Os

• Upgraded to Hive 3.1.3 for HCatalogIO. Users can still provide their own version of Hive. (Java) (Issue-19554).

New Features / Improvements

• Go SDK users can now use generic registration functions to optimize their DoFn execution. (BEAM-14347)
• Go SDK users may now write self-checkpointing Splittable DoFns to read from streaming sources. (BEAM-11104)
• Go SDK textio Reads have been moved to Splittable DoFns exclusively. (BEAM-14489)
• Pipeline drain support added for Go SDK has now been tested. (BEAM-11106)
• Go SDK users can now see heap usage, sideinput cache stats, and active process bundle stats in Worker Status. (BEAM-13829)

Breaking Changes

• The Go Sdk now requires a minimum version of 1.18 in order to support generics (BEAM-14347).
• synthetic.SourceConfig field types have changed to int64 from int for better compatibility with Flink's use of Logical types in Schemas (Go) (BEAM-14173)
• Default coder updated to compress sources used with BoundedSourceAsSDFWrapperFn and UnboundedSourceAsSDFWrapper.

Bugfixes

• Fixed Java expansion service to allow specific files to stage (BEAM-14160).
• Fixed Elasticsearch connection when using both ssl and username/password (Java) (BEAM-14000)

[2.39.0] - 2022-05-25

Highlights

• Watermark estimation is now supported in the Go SDK (BEAM-11105).
• Support for impersonation credentials added to dataflow runner in the Java and Python SDK (BEAM-14014).
• Implemented Apache PulsarIO (BEAM-8218).

I/Os

• JmsIO gains the ability to map any kind of input to any subclass of javax.jms.Message (Java) (BEAM-16308).
• JmsIO introduces the ability to write to dynamic topics (Java) (BEAM-16308).
  • A topicNameMapper must be set to extract the topic name from the input value.
  • A valueMapper must be set to convert the input value to JMS message.
• Reduce number of threads spawned by BigqueryIO StreamingInserts ( BEAM-14283).
• Implemented Apache PulsarIO (BEAM-8218).

New Features / Improvements

... (truncated)

Commits

• b95cc45 Set version for 2.40.0 RC2
• 313f039 Merge pull request #22007 from [release-2.40.0] Fix FlatMap numpy array bug
• 92a57ce Fix flatmap bug
• 52834cd Add failing test
• 882b608 Merge pull request #21944 from [cherry-pick][release-2.40.0][21941] Fix no ou...
• c85af81 Merge pull request #21940 from [21941] Fix no output timestamp case
• 6afcb5b Merge pull request #21938 from [cherry-pick][release-2.40.0] BigQueryIO: Addi...
• 12d0fe4 Merge pull request #21936 from [cherry-pick][release-2.40.0][Fixes #21927] Co...
• 91ae5be BigQueryIO: Adding the BASIC view setting to getTable request (#21879)
• 1cb7eb6 Merge pull request #21928 from [Fixes #21927] Compress (Un)BoundedSourceAsSdf...
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)