beam
beam copied to clipboard
apache
Apache Beam is a unified programming model for Batch and Streaming data processing.
### What happened? Beam vendors a few dependencies (grpc, guava). These vendored dependency jars have package namespace shaded (* -> org/apache/beam/vendor/*) so it won't leak classes. However this repackage does...
### What happened? To reproduce, run ```bash wget https://repo1.maven.org/maven2/org/apache/beam/beam-examples-multi-language/2.59.0/beam-examples-multi-language-2.59.0.jar java -jar beam-examples-multi-language-2.59.0.jar 33333 ``` there are folowing "INFO" level log with exception: ``` INFO: Could not load the TransformPayloadTranslator org.apache.beam.runners.dataflow.DataflowRunner$DataflowPayloadTranslator@32f0fba8...
The current MongoDB source is bounded, which means that we can't build streaming pipelines directly from MongoDB. MongoDB publishes changes in each collection through the oplog. Would it be possible...
Set processing Time timers and ProcessContinuations to be scheduled via a Real Time clock for a "quasi production" mode. Requires per-pipeline configuration to be plumbed into the Element manager to...
This pull request includes changes to the `expand` method in the `sdks/python/apache_beam/transforms/core.py` file to simplify the code and remove an unnecessary inner class. The most important changes include the removal...
### What happened? During the execution of BigQuery streaming job (STORAGE_API_AT_LEAST_ONCE method) utilizing `SyntheticUnboundedSource`, a difference between the number of generated messages and the actual number of rows persisted in...
Context: I was reading GCS Parquet files via SplittableDoFn and noticed that `ReadableFile#openSeekable` does not propagate any of the [gcs-connector options](https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/v2.2.25/gcs/src/main/java/com/google/cloud/hadoop/fs/gcs/GoogleHadoopFileSystemConfiguration.java#L66-L562) specified in my `core-site.xml` file. Particularly, I wanted to...
Addresses https://github.com/apache/beam/issues/32681 Following https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-HowtoupdatedependenciesthatareinstalledinPythoncontainerimages and https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-InstallingPythoninterpreters, updated the base image deps. ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: -...
Bumps [@octokit/request-error](https://github.com/octokit/request-error.js) to 5.1.1 and updates ancestor dependencies [@octokit/request-error](https://github.com/octokit/request-error.js), [@actions/github](https://github.com/actions/toolkit/tree/HEAD/packages/github) and [@octokit/rest](https://github.com/octokit/rest.js). These dependencies need to be updated together. Updates `@octokit/request-error` from 2.1.0 to 5.1.1 Release notes Sourced from @octokit/request-error's...
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot] avatar"){kind=avatar}
Bumps [github.com/linkedin/goavro/v2](https://github.com/linkedin/goavro) from 2.13.1 to 2.14.0. Release notes Sourced from github.com/linkedin/goavro/v2's releases. v2.14.0 What's Changed Update import to v2 in doc example by @loganrosen in linkedin/goavro#275 logicalTypes: fix alternative encoding...