Add new resources for Vertex AI Vector Search (formally known as, Vertex AI Matching Engine)

[shotarok]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

Add new resources for Vertex AI Matching Engine. The index and indexEndpoint API endpoints have create, patch, get, and delete methods. Therefore, they would be straightforward to implement.

On the other hand, the following API endpoints correspond to CRUD operations to deploy an index to an endpoint. Therefore, I put another new resource as an implementation idea in "New or Affected Resource(s)".

• Create: projects.locations.indexEndpoints/deployIndex
• Read: projects.locations.indexEndpoints/get
  • Note: The response object: IndexEndpoint has a list of DeployedIndex as deployedIndexes
• Update: projects.locations.indexEndpoints/mutateDeployedIndex
• Delete: projects.locations.indexEndpoints/undeployIndex

As a side note, Vertex AI Endpoints (projects.locations.endpoints) has similar methods: deployModel, undeployModel. That'd be great if both Matching Engine and Endpoint resources could have similar resources to deploy index or model to an endpoint.

New or Affected Resource(s)

• ~google_vertex_ai_index~
• ~google_vertex_ai_index_endpoint~
• google_vertex_ai_index_endpoint_deployed_index or google_vertex_ai_deployed_index

Potential Terraform Configuration

# Propose what you think the configuration to take advantage of this feature should look like.
# We may not use it verbatim, but it's helpful in understanding your intent.

References

• API References
  • https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.indexes
  • https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.indexEndpoints
  • https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.indexEndpoints/deployIndex
• https://github.com/hashicorp/terraform-provider-google/issues/9298 is a related issue

b/309602461

[mjpolak]

Hey, Is there any schedule on when it will be available? I can create indexes and endpoints now by google_vertex_ai_index and google_vertex_ai_index_endpoint But deploying index require now to implement custom code.

[shotarok]

I'm afraid I don't have any ongoing PR right now. I've been using the following gcloud commands for the deploy and undeploy operations.

• https://cloud.google.com/sdk/gcloud/reference/ai/index-endpoints/deploy-index
• https://cloud.google.com/sdk/gcloud/reference/ai/index-endpoints/undeploy-index

[mikeedjones]

Is it possible to add support for the Private service connect options, as documented https://cloud.google.com/vertex-ai/docs/matching-engine/match-eng-setup/private-service-connect

Understand its still in pre-GA.

Thanks!

[shotarok]

Thanks for letting me know about the new feature. It looks like both v1 and v1beta1 support privateServiceConnectConfig in a request body like below. After figuring out how to test a private service connect in an acceptance test of magic-modules, I can work on it.

• https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.indexEndpoints#privateserviceconnectconfig
• https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/projects.locations.indexEndpoints#privateserviceconnectconfig

[mikeedjones]

Thank you! Yes that's what we're using at the moment. The feature isn't supported in the python client library - at either creation or query time.

It looks like privateServiceConnect is deprecated on model Endpoints - is that likely to happen to index endpoints as well?

[shotarok]

It looks like enablePrivateServiceConnect is deprecated in the endpoint. The index endpoint also has the deprecated field. I believe privateServiceConnectConfig is the new alternative field to the deprecated field, although I can't know about any internal roadmap 🙂

[ruanspies]

I'm afraid I don't have any ongoing PR right now. I've been using the following gcloud commands for the deploy and undeploy operations.

Is the deploy and undeploy endpoint functionality within scope for resources associated with Vertex AI Matching Engine? Would be keen to collaborate on it.

[shotarok]

@ruanspies Yes, I put the endpoints for CRUD on the issue description. Because of the API endpoint design, I anticipate a lot of custom resource codes are necessary for implementing a resource like google_vertex_ai_index_endpoint_deployed_index . Please take a look at the description for the details.

As a side note, an index-related operations can take a few hours as their timeouts are set to 180. I'd recommend using the data resource: google_vertex_ai_index when running a test locally.

[BenLiyanage]

Definitely interested in support for modifying the deployed_index.

[melinath]

Remaining work here: Adding support for google_vertex_ai_index_endpoint_deployed_index or google_vertex_ai_deployed_index (or determining that they don't make sense in Terraform)

[Ludovic-Emo-Pyl-Tech]

Need definitely the capability to deploy an index on an endpoint with terraform.

[EduardJoy]

any updates for this?

[shotarok]

@mikeedjones The PR to support PSC was merged yesterday. The PSC will be available after a new version is released!

[EduardJoy]

Great, then the only issue that remains is the google_vertex_ai_index_endpoint_deployed_index resource

[davidcavazos]

Are there any updates on when the google_vertex_ai_index_endpoint_deployed_index is planned to be ready?

[davidcavazos]

In the meantime, here's a workaround using gcloud to deploy and undeploy the index.

resource "google_vertex_ai_index" "my_index" {
  ...
}

resource "google_vertex_ai_index_endpoint" "my_endpoint" {
  ...
}

module "gcloud_ai_index_endpoints_deploy_index" {
  source  = "terraform-google-modules/gcloud/google"
  version = "~> 3.0"

  create_cmd_body  = "ai index-endpoints deploy-index ${google_vertex_ai_index_endpoint.my_endpoint.id} --deployed-index-id=deployed_index --display-name=deployed_index --index=${google_vertex_ai_index.my_index.id} --project=${module.project_services.project_id} --region=${var.region}"
  destroy_cmd_body = "ai index-endpoints undeploy-index ${google_vertex_ai_index_endpoint.my_endpoint.id} --deployed-index-id=deployed_index --project=${module.project_services.project_id} --region=${var.region}"
}

[EduardJoy]

Any luck that someone might pick this up?

[Freezaa9]

Thanks @davidcavazos Any news about including this natively?

[SaschaHeyer]

Is there any update?

[NicolaSpreafico]

Any update on this?

[melinath]

Hey folks, this ticket has been forwarded to engineering for resolution. Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

[nicola-spreafico]

In the meantime, here's a workaround using gcloud to deploy and undeploy the index.

resource "google_vertex_ai_index" "my_index" {
  ...
}

resource "google_vertex_ai_index_endpoint" "my_endpoint" {
  ...
}

module "gcloud_ai_index_endpoints_deploy_index" {
  source  = "terraform-google-modules/gcloud/google"
  version = "~> 3.0"

  create_cmd_body  = "ai index-endpoints deploy-index ${google_vertex_ai_index_endpoint.my_endpoint.id} --deployed-index-id=deployed_index --display-name=deployed_index --index=${google_vertex_ai_index.my_index.id} --project=${module.project_services.project_id} --region=${var.region}"
  destroy_cmd_body = "ai index-endpoints undeploy-index ${google_vertex_ai_index_endpoint.my_endpoint.id} --deployed-index-id=deployed_index --project=${module.project_services.project_id} --region=${var.region}"
}

Thank you, Your workaround works in order to deploy the index, but I'm having an hard time retrieving the id of the deployed index, because being an asynchronous I should wait in any case after the gcloud command and then retrieve the final Id.

I hope the resource will be supported soon on the terraform provider so it will go smooth in terms of integrations

[slocoro]

Would also be interested in gcloud_ai_index_endpoints_deploy_index

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.