mlops-with-vertex-ai icon indicating copy to clipboard operation
mlops-with-vertex-ai copied to clipboard

Missing gcloud command or terraform code to create cloud build triggers

Open hilliao opened this issue 2 years ago • 0 comments

Users like to have a way to create cloud build triggers in the Terraform folder or with gcloud commands. I found tensorflow transform is using Apache beam runner in the following cloud build file.

# Compile the pipeline.
- name: '$_CICD_IMAGE_URI'
  entrypoint: 'python'
  args: ['build/utils.py',
          '--mode', 'compile-pipeline',
          '--pipeline-name', '$_PIPELINE_NAME'
          ]
  dir: 'mlops-with-vertex-ai'
  env: 
  - 'PROJECT=$_PROJECT'  
  - 'REGION=$_REGION'
  - 'MODEL_DISPLAY_NAME=$_MODEL_DISPLAY_NAME'
  - 'DATASET_DISPLAY_NAME=$_DATASET_DISPLAY_NAME'  
  - 'GCS_LOCATION=$_GCS_LOCATION' 
  - 'TFX_IMAGE_URI=$_TFX_IMAGE_URI' 
  - 'BEAM_RUNNER=$_BEAM_RUNNER'
  - 'TRAINING_RUNNER=$_TRAINING_RUNNER'
  id: 'Compile Pipeline'
  waitFor: ['Local Test E2E Pipeline']

So far I gather an example from a notebook's section. But I can't find how to create the cloud build trigger.

  1. Run the training pipeline using Vertex Pipelines Set the pipeline configurations for the Vertex AI run os.environ["DATASET_DISPLAY_NAME"] = DATASET_DISPLAY_NAME os.environ["MODEL_DISPLAY_NAME"] = MODEL_DISPLAY_NAME os.environ["PIPELINE_NAME"] = PIPELINE_NAME os.environ["PROJECT"] = PROJECT os.environ["REGION"] = REGION os.environ["GCS_LOCATION"] = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}" os.environ["TRAIN_LIMIT"] = "85000" os.environ["TEST_LIMIT"] = "15000" os.environ["BEAM_RUNNER"] = "DataflowRunner" os.environ["TRAINING_RUNNER"] = "vertex" os.environ["TFX_IMAGE_URI"] = f"gcr.io/{PROJECT}/{DATASET_DISPLAY_NAME}:{VERSION}" os.environ["ENABLE_CACHE"] = "1"

hilliao avatar Apr 03 '22 17:04 hilliao