beam
beam copied to clipboard
apache
Apache Beam is a unified programming model for Batch and Streaming data processing.
### What needs to happen? Currently Beam CI still uses Java8. There were build plugins and dependencies start dropping java8 support (e.g. the docker plugin beam used to use and...
### What happened? Installing apache-beam into a clean environment now shows following logs: ``` pip install --no-cache-dir --pre apache-beam[gcp]==2.56.0 ... Collecting grpcio-status>=1.33.2 (from google-cloud-pubsub=2.1.0->apache-beam[gcp]==2.56.0) Downloading grpcio_status-1.64.1-py3-none-any.whl.metadata (1.1 kB) Downloading grpcio_status-1.64.0-py3-none-any.whl.metadata...
### What would you like to happen? All cross lang transformations that are implemented in java that have a python wrapper that use the old ExternalTransform API such as KafkaIO...
### What happened? Beam has PubsubMessage PubSubMessage PubsubMessages. The use is inconsistent and confusing. It would be good to use a standard naming scheme. (PubSubMessage seems to be the correct...
Confluent Schema Registry (CSR) 6 introduces support for other Schema types like Protobuf and JSON Schema. We should adapt KafkaIO to obtain the type of Schema and configure correctly its...
### What would you like to happen? The Storage API BigQuery sink already handles rows that do not conform to schema and forwards them to the failed-rows PCollection. We should...
### What happened? Unable to trigger early firings with a session window. Tried triggers: ```go windowedUpdates := beam.WindowInto(s, window.NewSessions(30*time.Second), propUpdates, beam.Trigger(trigger.Repeat(trigger.AfterCount(500))), beam.Trigger(trigger.Repeat(trigger.AfterProcessingTime().PlusDelay(60*time.Second))), beam.PanesDiscard(), ) ``` ```go windowedUpdates := beam.WindowInto(s, window.NewSessions(30*time.Second),...
### What happened? When importing Avro files that have schemas with the field type: ```json {"type": "long", "logicalType": "timestamp-millis"} ``` an attempt to convert the collection to Beam Schemas (beam.Row)...
### What happened? I'm trying to inspect the workflow graph generated by Beam without submitting the job to Dataflow. So I used the following command: ``` python -m wordcount --runner...
Hello! Thank you for all the hard work, it is very much appreciated. Is there a matrix that maps elastic version to java-io-elasticsearch version? I noticed [rest client 7.9.2](https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/build.gradle) as...
