[Question]: When do workflows get published to gs://dataflow-templates

[shadiramadan]

Related Template(s)

BigQueryToBigtable

What feature(s) are you requesting?

bigquery-to-bigtable/README.md

When do templates get promoted / published to gs://dataflow-templates ? I think it would be beneficial to track releases in this repository and have the releases result in a new folder/update for the templates in GCS.

I went crazy not finding this template- I was going to use the avro to bigtable one and export from bigquery to avro. It's way easier when the templates are premade!

[shadiramadan]

Side note- would it not make more sense for these images to be hosted in a google artifact registry and for the dataflow CLI commands to use that?

[zhoufek]

Hi, Shadi!

As of right now, that template is unreleased. We could probably more clearly mark that on GitHub. As long as it is unreleased, you'll need to follow the steps in the readme to run it.

Currently, I don't think there's an ETA on when it will be available. We normally try to have some integration testing set up before releasing anything, and we're currently in a bit of a transition period in that regard. All those tests are internal to Google, but we're trying to rewrite them all on GitHub, and I think @billyjacobson is waiting until then to write the tests. That'll also tie it to when our releases are also managed from GitHub, so I think the earliest we can expect is 6 - 9 months.

Side note- would it not make more sense for these images to be hosted in a google artifact registry and for the dataflow CLI commands to use that?

Personally, I would love to do that and reserve the current commands for development purposes. I don't know if has been brought up internally, but it is certainly technically possible. I'm just not sure what would be required from a policy standpoint or if our team can feasibly keep up with those requirements. I'll try to bring it up with the team, but I can't give any guarantees on if this will happen or when it will happen.

[zhoufek]

I think it would be beneficial to track releases in this repository and have the releases result in a new folder/update for the templates in GCS.

Sorry, missed this in the first reply.

I think that the plan was to start doing this regularly once we moved managing releases to GitHub, but I think it makes sense to publish release changes here. The only downside is that those releases may not contain all of the repo changes, since there's always a delay in adding a template and getting it officially released through the Cloud Console.