Regional TCP LBs should use Instance Groups rather than Target Pools

[ljfranklin]

The RelInt team discovered issues with failed HTTP requests during a deploy when Target Pools were used. The current theory by Google engineers is that switching to Instance Groups instead will fix this issue. Looks like y'all have a story to make this change in bbl as well.

The Terraform docs only mention Target Pools as the valid target for forwarding rules. However the API docs seem to indicate that an Instance Group ID is a valid value for target.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[nwmahoney]

I think it is still an open question how we will achieve this suggested change in terraform/GCP, so once we solve it in either product (terraforming-gcp or bbl), we will be sure to follow through to the other product quickly.

[crhntr]

Hello @ljfranklin, I've gathered a bit of context on this issue over the last few days while working on solving it for bbl. It looks like this will be a feature in the future but Google cloud does not support this.

• In order to use instance groups, they must be linked to a backend_service.
• Only internal regional backend services "can" do network load balancing.
  • Backend services can be global or regional.
  • Backend services "can" have internal or external load balancing schema.
• After updating terraform-provider-google to support modification of the LoadBalancerScheme, I found that google returns the following error:

  * google_compute_region_backend_service.router-lb-backend-service: 1 error(s) occurred:
  
  * google_compute_region_backend_service.router-lb-backend-service: Error creating backend service: googleapi: Error 400: Invalid value for field 'resource.loadBalancingScheme': 'EXTERNAL'. Backend Service based Network Load Balancing is not yet supported., invalid
  

Using instance groups rather than target pools is blocked on Google releasing the external backend services feature. Since the error message suggests that Google Cloud recognizes what we want to do but claims "it is not yet supported", we may be able to reach out to our friends at Google to see if the feature is implemented but not generally available and if so request access in some alpha form.

*Links to supporting Google/Terraform docs are in the (private) PivotalTracker story. #161389908

[ljfranklin]

I was under the impression that RelInt had seen this working. cc @staylor14