Enable different CORS handling for different endpoints

[nrabinowitz]

We are using ESPv2 to manage endpoints for our backend API, including dozens of endpoints provided by different services in our cluster. We currently have CORS set to accept only a small number of domains (using --cors_preset=cors_with_regex). However, some of our endpoints represent a "public" API, meant for external use, and we'd like to provide the Access-Control-Allow-Origin: * header for those endpoints.

Is there any way to configure ESPv2 for this scenario? As far as I can tell, our choices are to turn off CORS handling in ESPv2, requiring us to add CORS support to every underlying BE service, or to move all endpoints to the more permissive setting, neither of which is an attractive option.

[qiwzhang]

ESPv2 is using Envoy. Currently, Envoy always adds the response header Access-Control-Allow-Origin as the Origin header from the request. It will not add response header Access-Control-Allow-Origin: *

[nrabinowitz]

Sorry, I was basing my wording on the docs:

Sets the response headers to the following values:

Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, PATCH, DELETE, OPTIONS
Access-Control-Allow-Headers: DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization
Access-Control-Expose-Headers: Content-Length,Content-Range
Access-Control-Max-Age: 1728000

What I meant was that we want specific endpoints to allow all origins. Is this possible?

[qiwzhang]

I see. You want to be able to define different CORS policy for different backends. Is that correct?
Currently, these CORS flags can only define one CORS policy, and this policy applies to all backends.

[nrabinowitz]

That's correct. Thanks for validating my assumption here - please consider this a feature request in that case 😁.

[sgammon]

@nrabinowitz worth noting that ESP is designed to be 1:1 with each API interface you export. so if you run an ESP instance per-service, you can apply CORS settings per service. i'm not sure if you're using OpenAPI or gRPC, but in gRPC's case (we use gRPC), you can mix and match which gRPC services map to which Endpoints services.

[nrabinowitz]

@sgammon Interesting. We might have an odd case - we're using ESP as a unified API layer for multiple backend services on a single externally-facing domain. In this case, even per-service CORS settings wouldn't be sufficient, as we expose both "public" and internal endpoints (internal here means "not part of our documented public API", rather than truly private) from the same service, and would like different CORS settings for these different endpoints.

[sgammon]

@nrabinowitz i hear you, we have the same setup. here's how we ended up doing this:

  client
     |
     |  <-- supports HTTP/1.1 (JSON transcoding), HTTP2 (grpc, grpc web)
     |
/-----------\
| GCLB (L7) |---|
\-----------/   | 
                |
            ------------
            |   envoy  |
            ------------
                  |
         -------------------
         |        |        |
   --------    --------    ------------------
   | esp  |    | esp  |    | other services |
   --------    --------    ------------------
       |          |
   --------    --------
   | grpc |    | grpc |
   --------    --------

some notes about this architecture:

• "envoy" is a vanilla Envoy agent running at latest on alpine. similar to ESP, but does not have ESP's addons. on the other hand, you get full access to envoy's extensive config.
• in our case, we use ASM inside GKE, so that's how services talk to each other. we didn't want to enforce API keys and quota in between services, so services don't traverse ESP. we only use ESP for egress/ingress with the public internet via LBs.
• as far as i know this pattern works with openAPI too, i'm not sure which you're using, openAPI or grpc.

drawbacks of this architecture:

• you have to run a lot of ESPs, it's a three-layer architecture by nature
• tracing can be tricky to get right
• you have a bunch of YAML and other configs to keep in sync
• sometimes it's hard to diagnose what layer is giving you an error

benefits of this architecture:

• complete control over ingress, routing, headers, SSL termination, etc (+CORS) at the edge layer
• easy support for version aliasing or routing (i.e. /api/latest/... -> /api/v1/... under the hood)
• CORS and other stuff can be applied per-service
• API key enforcement via Google's tooling still works great and can be applied per-service
• able to do fancy API stuff like websockets and still share a domain/URL endpoint

in terms of documentation, that got complex for us, but we ended up generating open API specs from our protos, which we could then use with apigee portal, apimatic, swagger UI, and so on.

this is just what we did to solve that problem, it worked for us, ymmv.

[sgammon]

(one more note. we found envoy's CORS support to be much more flexible with direct control at the edge, rather than via ESP's flags. since it's only the browser that enforces CORS, it works fine to append those headers after ESP has finished processing, and leave CORS off from ESP's perspective entirely)