mlops-with-vertex-ai
mlops-with-vertex-ai copied to clipboard
Missing gcloud command or terraform code to create cloud build triggers
Users like to have a way to create cloud build triggers in the Terraform folder or with gcloud commands. I found tensorflow transform is using Apache beam runner in the following cloud build file.
# Compile the pipeline.
- name: '$_CICD_IMAGE_URI'
entrypoint: 'python'
args: ['build/utils.py',
'--mode', 'compile-pipeline',
'--pipeline-name', '$_PIPELINE_NAME'
]
dir: 'mlops-with-vertex-ai'
env:
- 'PROJECT=$_PROJECT'
- 'REGION=$_REGION'
- 'MODEL_DISPLAY_NAME=$_MODEL_DISPLAY_NAME'
- 'DATASET_DISPLAY_NAME=$_DATASET_DISPLAY_NAME'
- 'GCS_LOCATION=$_GCS_LOCATION'
- 'TFX_IMAGE_URI=$_TFX_IMAGE_URI'
- 'BEAM_RUNNER=$_BEAM_RUNNER'
- 'TRAINING_RUNNER=$_TRAINING_RUNNER'
id: 'Compile Pipeline'
waitFor: ['Local Test E2E Pipeline']
So far I gather an example from a notebook's section. But I can't find how to create the cloud build trigger.
- Run the training pipeline using Vertex Pipelines Set the pipeline configurations for the Vertex AI run os.environ["DATASET_DISPLAY_NAME"] = DATASET_DISPLAY_NAME os.environ["MODEL_DISPLAY_NAME"] = MODEL_DISPLAY_NAME os.environ["PIPELINE_NAME"] = PIPELINE_NAME os.environ["PROJECT"] = PROJECT os.environ["REGION"] = REGION os.environ["GCS_LOCATION"] = f"gs://{BUCKET}/{DATASET_DISPLAY_NAME}" os.environ["TRAIN_LIMIT"] = "85000" os.environ["TEST_LIMIT"] = "15000" os.environ["BEAM_RUNNER"] = "DataflowRunner" os.environ["TRAINING_RUNNER"] = "vertex" os.environ["TFX_IMAGE_URI"] = f"gcr.io/{PROJECT}/{DATASET_DISPLAY_NAME}:{VERSION}" os.environ["ENABLE_CACHE"] = "1"