GCP support

[pimvernooij]

We intend to support GCP as a cloud environment, and this ticket is inteded to specify 'how' we are going to support it.

First read this page: https://docs.machcomposer.io/tutorial/gcp/step-3-setup-gcp.html

Please use this thread to discuss/propose.

To Do:

• [x] Add support in MACH yaml to support GCP cloud
• [x] Add support in commercetools provider for Google Cloud Function api extension
• [x] Add terraform templates to MACH composer, to support GCP resources
• [x] Add support for storing terraform state remotely in GCP
• [ ] Implement DNS record management via Google Cloud DNS (?)
• [x] Implement routing using Google API Gateway
• [ ] Implement GCP Secrets Manager as 'native' secrets management solution (in component cookiecutter)
• [ ] Reference implementation of a CloudRun and CloudFunctions serverless function (in component cookiecutter)
• [ ] Create reference Google Cloud account/project setup where the architecture runs in
• [ ] Ideally: extend component bootstrapper and component cookiecutter to include Google Cloud setup
• [ ] Unit/integration testing
• [ ] Expand documentation with GCP

Proposed services to use in GCP

• Remote Terraform state: Google Cloud Storage
• DNS: Google Cloud DNS
• HTTP Routing / HTTP Gateway: multiple candidates (API Gateway, Apigee, Cloud Endpoints), but API Gateway seems to be the designated 'future' Gateway service in GCP? See quickstart. With beta support for it in terraform.
• Secrets management: GCP secrets manager.
• Serverless functions: Google Cloud Functions
• Serverless Docker containers: Google CloudRun

Interesting read: https://dev.to/didil/gcp-api-gateway-demo-with-terraform-go-cloud-run-3o9e

For all services, we need to take into account that it should be deployed through terraform completely. Though solutions exist when there is no 'full blown' terrafrom support, in which case you could fall back to a CLI. We recently did this to implement Apollo Studio support: #78

account setup in GCP

We need to determine what 'project structure' will be used in GCP. In other clouds we use multiple resource groups (Azure) and Accounts (AWS) to provide multi-tenancy/platform partitioning between sites. The same should be achieved with GCP.

This is TBD.

[Nayels]

I like the initial setup and approach. Nicely done @pimvernooij.

I've got the most questions about the account setup in GCP.

• The resource groups that are created now on Azure. How do they communicate with each other?
• Could you give me an example of products/applications per resource group? AKA what is the purpose of THAT specific resource group I've tried opening the share infra module for Azure with no success.

The reason why I'm asking this is because, if we're going to use multi-tenancy, we should also add Shared VPC to the proposal for internal communication between services.

[tleguijt]

Hi @Nayels ,

The resource groups that are created now on Azure. How do they communicate with each other?

They don't. Or at least; the only link between resource groups is only between 'shared' and site-specific;

• Registering a DNS name on the DNS zone settings in the shared resource group
• Pulling a ZIP file containing function app code from a blob container on the shared resource group.

I will give you access to the shared infra module as well

[pimvernooij]

Decision: we go with Google API Gateway unless there is a good reason not to :).

[tleguijt]

We can now have a component defining an API config like so:

resource "google_api_gateway_api_config" "api_cfg" {
  provider = google-beta
  api = var.gcp_endpoint_public.api_id
  api_config_id = "gcp-api-cfg"

  openapi_documents {
    document {
      path = "spec.yaml"
      contents = base64encode(data.template_file.spec.rendered)
    }
  }
  lifecycle {
    create_before_destroy = true
  }
}

But from a MACH perspective, we need to be able to combine different 'Gateway API configs' into one; so that all traffic to /componentA/* goes to a specific API configuration and traffic to /componentB/* would be routed to the API configuration of another component.

Is this something that can be achieved using API gateway? If we have solved that, we can focus on hooking up the API gateway to a custom domain.

[okkevandereijk]

@Nayels and I today investigated in what extend how we can use the Google API gateway for usage in the MACH composer to combine the differente API configs and have discussed this with @pimvernooij. We concluded the following:

• We have tested an approach for using an API gateway gateway (in manner an additional Google API Gateway) with references to gateways to be re routed. Because it looks like the Google API gateway is bounded to the usage of defined Open API Spec we searched for a way to do it there.
• We found out that the Open API Spec has limited/no ways to use a wildcard options, for combining the different API Gateways. We tried the APPEND_PATH_TO_ADDRESS possibilities, but this doesn't do what we are looking for.

So basically to get this construction up-and-running you need to explicitly define all the endpoints and definitions in the Open API Spec in this API gateway gateway of all the different components. The only way to have this practically working is to automate the creation of a combined Open API spec, based on all the different Open API specs of the components.

[Nayels]

To add to the comment of @okkevandereijk The common API Gateway spec would look something like this:

swagger: '2.0'
info:
  title: MACH API
  description: MACH GCP API
  version: 0.1.0
schemes:
  - https
produces:
  - application/json
paths:
  /api/healthcheck:
    get:
      summary: Healthcheck endpoint
      operationId: healthcheck
      x-google-backend:
        address: https://api-component-4m55fqmy.nw.gateway.dev/api/healthcheck
      responses:
        '200':
          description: OK
  /api/payment:
    get:
      summary: Payment endpoint
      operationId: healthcheck
      x-google-backend:
        address: https://api-payment-4m55fqmy.nw.gateway.dev/api/payment
      responses:
        '200':
          description: OK

[okkevandereijk]

However based on those findings, we concluded that there seems to be not really a reason to create an API gateway gateway (additional API Gateway) to combine the different API gateways, if you still need to combine all the different Open API specs.

We propose to use the following approach:

• We create only one API Gateway per GCP project (so for every website running in the MACH composer)
• Each component will have his own Open API Spec
• The Open API specs for each component should be combined in the Open API Spec of the API Gateway

For now, we will use an approach in which those specs needs to be manually maintained, it would be great if this can be automated in the future.

[tleguijt]

We've been able to stitch together the api specs of several components into one. So new requirement for components with certain endpoints is that they have to define an output which MACH can use to read the API spec from.

Next step is to support custom domains. For this we need to put a load balancer in front of it. A couple of useful links are described at https://docs.machcomposer.io/tutorial/gcp/step-3-setup-gcp.html#http-routing-api-gateway and the following can also be useful: https://cloud.google.com/community/tutorials/modular-load-balancing-with-terraform