Support for flex template runtime environment options

[n-oden]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

The projects.locations.flexTemplates.launch API used by google_dataflow_flex_template takes a LaunchFlexTemplateParameter object as its payload. LaunchFlexTemplateParameter has an environment field that takes a FlexTemplateRuntimeEnvironment object, and this is where callers can specify standard dataflow job options such as the region/zone, the number of workers, the network/subnet etc etc.

Unfortunately, the google_dataflow_flex_template_job resource in the google-beta provider does not currently expose the environment field, so it is not possible to pass any of these options to the job. As a workaround they can be specified in the parameters section, but this requires that the param names be listed in the job's metadata.json file and any parameter not listed there cannot be passed.

New or Affected Resource(s)

• google_dataflow_flex_template_job

Potential Terraform Configuration

resource "google_dataflow_flex_template_job" "big_data_job" {
  provider                = google-beta
  name                    = "dataflow-flextemplates-job"
  container_spec_gcs_path = "gs://my-bucket/templates/template.json"
  parameters = {
    inputSubscription = "messages"
  }
  environment = {
    num_workers = 8
    machine_type = "n1-standard-8"
    enable_streaming_engine = true
    additional_user_labels = {
      billing_component = "dataproc"
    }
  }
}

b/374162542