liferoad

Throw RuntimeException with the resource id when IOException happens in FileBasedSource

1

Throwing the error that contains the resource id could help users identify which file could cause issues. ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate...

[Task]: Improve how to handle the Dataflow-specific option `impersonateServiceAccount` for Beam Java

1

### What needs to happen? `impersonateServiceAccount` should be kept when submitting Dataflow jobs but should be removed when creating Dataflow workers per [the design](https://docs.google.com/document/d/13KRYiq5JAcs-leznzXI_knvqp7ud0u3YASVqK-yMeQw/edit#heading=h.18gu8586i6j1). To fix this, #30283 put a...

### What happened? # Test code ```python import apache_beam as beam import datetime CSV_DATA = r"""a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r 1,"text",1,21,5945023,376974,0,0,0,1,2,0,4,,,,,, """ filename = '/tmp/input.csv' with open(filename, 'w') as f: f.write(CSV_DATA) # less than...

Retry when the BQ job error can be retried

1

Address #30513 for the flaky BQ test. From https://ge.apache.org/s/rwjhs7f4gc2ee/console-log/task/:sdks:python:test-suites:direct:py39:postCommitIT?anchor=424&page=1, ``` [1mERROR  apache_beam.runners.common:bundle_processor.py:237 BigQuery job beam_bq_job_COPY_AUTOMATIC_JOB_NAME_COPY_STEP_76772e260e0c46cf96f0b94402c2595d_2af7745dca2d9060a22b202c77bbc2d8 failed. Error Result: [while running 'write/BigQueryBatchFileLoads/ParDo(TriggerCopyJobs)/ParDo(TriggerCopyJobs)'] ``` It looks like we should retry when...

Remove GlobalWindows for unbounded pcoll without a default global window

2

Fixes #25598 ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the appropriate issue in your description...

### What happened? More details are here: https://stackoverflow.com/questions/64526500/using-writetobigquery-file-loads-in-a-streaming-pipeline-just-creates-a-lot-of-t ### Issue Priority Priority: 3 (minor) ### Issue Components - [X] Component: Python SDK - [ ] Component: Java SDK - [...

### What would you like to happen? FileIO supports https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.Write.html#withAutoSharding, which usually works well to split large bundles to smaller ones and then write the shards to files. However, in...

Update direct_runner.py

1

Improve the error message for DirectRunner. ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the appropriate...

Update the python deps caused by numpy 2.0

2

Addresses https://github.com/apache/beam/issues/32681 Following https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-HowtoupdatedependenciesthatareinstalledinPythoncontainerimages and https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-InstallingPythoninterpreters, updated the base image deps. ------------------------ Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: -...