DataflowTemplates icon indicating copy to clipboard operation
DataflowTemplates copied to clipboard

Support for export specified tables from Cloud Spanner to Cloud Storage Avro template

Open shashank-google opened this issue 4 years ago • 2 comments

Currently export dumps entire database. Can we modify this template to include tables argument as well ? Such that only specified tables are exported our from a particular database instance.

shashank-google avatar Jan 20 '21 22:01 shashank-google

This would be a nice addition!

We currently have an internal fork with customizations to exclude tables from a spanner to gcs export (avro format), which takes time to maintain and keep up to date with upstream changes. It would great if the spanner export/import templates supported a UDF to filter tables out of the archive.

kb avatar Jan 26 '21 16:01 kb

Based on recent commits, it looks like this feature has been developed:

  • https://github.com/GoogleCloudPlatform/DataflowTemplates/commit/43aab86d5d043703d55dbddf6c57059945f01742

But it looks like it's currently not available for use. Any information on when this would be released?

peterssonnick avatar Aug 18 '21 20:08 peterssonnick

As mentioned by @peterssonnick, this was already added. The support lagged a little bit due to the release taking a couple of days, but is already available.

bvolpato avatar Feb 21 '23 02:02 bvolpato