beam
beam copied to clipboard
apache
Apache Beam is a unified programming model for Batch and Streaming data processing.
Add option to enable Kafka Read with redistribute afterwards to increase paralellism ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:...
Address #30513 for the flaky BQ test. From https://ge.apache.org/s/rwjhs7f4gc2ee/console-log/task/:sdks:python:test-suites:direct:py39:postCommitIT?anchor=424&page=1, ``` [1m[31mERROR [0m apache_beam.runners.common:bundle_processor.py:237 BigQuery job beam_bq_job_COPY_AUTOMATIC_JOB_NAME_COPY_STEP_76772e260e0c46cf96f0b94402c2595d_2af7745dca2d9060a22b202c77bbc2d8 failed. Error Result: [while running 'write/BigQueryBatchFileLoads/ParDo(TriggerCopyJobs)/ParDo(TriggerCopyJobs)'] ``` It looks like we should retry when...
**Please** add a meaningful description for your change here ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ]...
This PR closes #31335 via refactoring BigQueryIO's assignment of the `_CHANGE_SEQUENCE_NUMBER` pseudo column from a long to fixed hex string. All BigQuery related tests for `./gradlew :sdks:java:io:google-cloud-platform:postCommit` and `./gradlew :sdks:java:io:google-cloud-platform:build`...
Add additional state tests for different coders. There's been a report of byte buffers becoming truncated or corrupted after being written to state. ------------------------ Thank you for your contribution! Follow...
When streaming inserts into BigQuery using BigQueryIO.Write, if there is an error other than a row insertion error (e.g. an IOException), BigQueryIO assumes this must be a rate limit error,...
The PreCommit Java Debezium IO Direct is failing over 50% of the time. Please visit https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Debezium_IO_Direct.yml?query=is%3Afailure+branch%3Amaster to see all failed workflow runs. See also Grafana statistics: http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1&viewPanel=3&var-Workflow=PreCommit%20Java%20Debezium%20IO%20Direct
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "github-actions[bot] avatar"){kind=avatar}
Reintroduce Flink 1.18 support. ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the appropriate issue in...
Bumps com.gradle.enterprise from 3.17.2 to 3.17.4. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a...
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot] avatar"){kind=avatar}