Support for `wait_for_crds` in `google_gke_hub_feature`

[brettcurtis]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Description

The plan fails when building MCS clusters and utilizing the k8s CRD: MultiClusterService in subsequent workflows since the CRDs are not fully installed yet.

│ Error: API did not recognize GroupVersionKind from manifest (CRD may not be installed)
│ 
│   with module.kubernetes_istio.kubernetes_manifest.istio_gateway_mcs[0],
│   on .terraform/modules/kubernetes_istio/regional/main.tf line 320, in resource "kubernetes_manifest" "istio_gateway_mcs":
│  320: resource "kubernetes_manifest" "istio_gateway_mcs" {
│ 
│ no matches for kind "MultiClusterService" in group "networking.gke.io"

It would be nice if this was managed at the resource vs. some W/A on the customer side.

New or Affected Resource(s)

• google_gke_hub_feature

Potential Terraform Configuration

No response

References

No response

[SarahFrench]

This looks like an instance of this : https://github.com/hashicorp/terraform-provider-kubernetes/issues/1782

I'll transfer the issue to the hashicorp/terraform-provider-kubernetes repo

[arybolovlev]

Hi @brettcurtis,

This is expected behavior due to how Terraform and the kubernetes_manifest resource are designed. CRD that handles the resource that you want to create should be installed first.

[brettcurtis]

Thanks for looking. I was talking about the google_gke_hub_feature resource in the google provider. I'm asking for the feature request there if possible. Maybe we can transfer it back @SarahFrench ?

When you enable MCS:

resource "google_gke_hub_feature" "multi_cluster_service_discovery" {
  name     = "multiclusterservicediscovery"
  location = "global"
  project  = local.global_infra.project_id

}

I believe it installs some CDRs as a result (maybe they are part of multiclusteringress I have both). I was simply hoping we could wait for those to be deployed before moving forward. Also just to point out my manifest workflow runs after this workflow has completed successfully so it must take at least a couple minutes for the CRDs t be fully installed on the backend after the resource completes.

The kubernetes_ingress resource has something like what I'm looking for here, wait_for_laod_balancer.

[arybolovlev]

manifest workflow runs after this workflow has completed successfully

Does this all happen within a single Terraform run?

[brettcurtis]

No, different Terraform runs.

[arybolovlev]

Thank you! When you re-run the one that executes kubernetes_manifest, does it succeed?

[brettcurtis]

Yes it does, usually less than a minute later, so it's quick.

[brettcurtis]

I think I have a theory around this looking into the logs a bit more. The gke_hub_feature for "multiclusterservicediscovery" doesn't depend on cluster or any other hub resources. For example this through line 303

So I suspect it's just an async process that looks for clusters that should have the feature and applies the CRDs. I'm going to test if a explicit dependency will help "speed up" the process. The idea being it may create the CRDs on apply and completion.

EDIT I tested the explicit dep, and it didn't help.

[github-actions[bot]]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

[brettcurtis]

Not stale