[Bug]: There have an error in 'WriteToBigQuery' after updating to 2.69

[ZwxwZ]

What happened?

For the Beam Yaml, there have an error in WriteToBQ after updating to 2.69,

ValueError: Unknow output 'error' at line 221: WriteResponseToBigQuery only has outputs []

look like WriteResponseToBigQuery update some features, to avoid the bug after using latest version, I want to fixed the version(2.68) of the Beam yaml, how to do this operation?

Issue Priority

Priority: 0 (outage / urgent vulnerability)

Issue Components

• [x] Component: Python SDK
• [ ] Component: Java SDK
• [ ] Component: Go SDK
• [ ] Component: Typescript SDK
• [ ] Component: IO connector
• [x] Component: Beam YAML
• [ ] Component: Beam examples
• [ ] Component: Beam playground
• [ ] Component: Beam katas
• [ ] Component: Website
• [ ] Component: Infrastructure
• [ ] Component: Spark Runner
• [ ] Component: Flink Runner
• [ ] Component: Samza Runner
• [ ] Component: Twister2 Runner
• [ ] Component: Hazelcast Jet Runner
• [x] Component: Google Cloud Dataflow Runner

[Abacn]

Beam 2.69 changed WriteToBigQuery implementation to Managed IO. It seems neither "Unknow output" nor "WriteResponseToBigQuery" are part of Beam SDK (search returns no result). What direct dependency are you using?

[ZwxwZ]

Beam 2.69 changed WriteToBigQuery implementation to Managed IO. It seems neither "Unknow output" nor "WriteResponseToBigQuery" are part of Beam SDK (search returns no result). What direct dependency are you using?

that error come from GCP dataflow console, we're using yaml dataflow, and get this error after version update to 2.69, not sure what is direct dependency ?

[liferoad]

@ZwxwZ can you share your YAML file?

[ZwxwZ]

can't share the whole yaml, but I can share the part which have error

 - name: Writetobq
   type: WriteToBigQuery
   inptu: MapUUIDForResponse
   config:
     table:"xxx"
     create_disposition: "CREATE_NEVER"
     write_disposition: "WRITE_APPEND"
     num_steams: "1"
    error_handling:
     output: error

error fixed by removing the

   error_handling:
     output: error

I am more interested in knowing if there is a method with a fixed version, because using the latest version automatically each time runs the risk of such errors.

[liferoad]

My guess is https://github.com/apache/beam/pull/36937. cc @ahmedabu98