beam icon indicating copy to clipboard operation
beam copied to clipboard

[Feature Request]: Add with .withFormatFunction() equivalent to Python SDK BigQueryIO

Open chishankar-work opened this issue 1 year ago • 0 comments

What would you like to happen?

Support customization for dynamic destinations when using Python SDK, BigQueryIO to allow users to embed destination tables without writing it to BigQuery.

A current workaround would to add destionation_table_name as a dummy column in the TableRow object and then use then set ignore_unkown_columns=True and then pass a callable table=getTableDestinationFn that gets called on each element and returns the destination_table_name.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

  • [X] Component: Python SDK
  • [ ] Component: Java SDK
  • [ ] Component: Go SDK
  • [ ] Component: Typescript SDK
  • [ ] Component: IO connector
  • [ ] Component: Beam YAML
  • [ ] Component: Beam examples
  • [ ] Component: Beam playground
  • [ ] Component: Beam katas
  • [ ] Component: Website
  • [ ] Component: Spark Runner
  • [ ] Component: Flink Runner
  • [ ] Component: Samza Runner
  • [ ] Component: Twister2 Runner
  • [ ] Component: Hazelcast Jet Runner
  • [ ] Component: Google Cloud Dataflow Runner

chishankar-work avatar Feb 15 '24 17:02 chishankar-work