503 error when deploying to App Engine

[maroux]

Hi

I'm trying to deploy esp in the following configuration:

ESPv2 - gcr.io/endpoints-release/endpoints-runtime-serverless:2.10.0 deployed to App Engine flex with env:

ESPv2_ARGS  = "^++^--cors_preset=cors_with_regex++--cors_allow_origin_regex=...++--enable_debug"
ENDPOINTS_SERVICE_NAME = "<service>-dot-<project>.appspot.com"

Endpoints config: Copy of sample with these changes:

-host: "YOUR-PROJECT-ID.appspot.com"
+host: "<service>-dot-<project>.appspot.com"

-    x-google-audiences: "YOUR-CLIENT-ID"
+    x-google-audiences: "<actual IAP client id>"

+  auth0_jwk:
+    authorizationUrl: ""
+    flow: "implicit"
+    type: "oauth2"
+    x-google-issuer: "https://<tenant>.auth0.com/"
+    x-google-jwks_uri: "https://<tenant>.auth0.com/.well-known/jwks.json"
+    x-google-audiences: "https://<esp service>-dot-<project>.appspot.com/"
+
+x-google-backend:
+  address: "https://<service>-dot-<project>.appspot.com"
+  jwt_audience: "<IAP oauth client id>"
+  protocol: h2
+
+
+x-google-endpoints:
+  - name: <service>-dot-<project>.appspot.com

When I hit the ESP service with appropriate JWT token, it fails with a 503 error:

HTTP/2 503 
date: Fri, 29 May 2020 23:15:26 GMT
content-type: text/plain
content-length: 91
server: envoy
via: 1.1 google
alt-svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

upstream connect error or disconnect/reset before headers. reset reason: connection failure

In ESP logs, I see:

D0529 22:30:59.101 27 envoy] [27][connection][external/envoy/source/common/network/connection_impl.cc:504] [C21] raising immediate error
...
D0529 22:30:59.101 27 envoy] [27][router][external/envoy/source/common/router/router.cc:1018] [C19][S9973334829757176186] upstream reset: reset reason connection failure
...
D0529 22:30:59.101 27 envoy] [27][http][external/envoy/source/common/http/conn_manager_impl.cc:1475] [C19][S9973334829757176186] Sending local reply with details upstream_reset_before_response_started{connection failure}

In Produced API logs, I see:

jsonPayload: {
  api_method: "1.<service>_dot_<project>_appspot_com.Echo"   
  api_name: "1.<service>_dot_<project>_appspot_com"   
  api_version: "1.0.0"   
  client_ip: "172.17.0.5"   
  http_method: "POST"   
  http_response_code: 503   
  location: "us-central1-b"   
  log_message: "1.<service>_dot_<project>_appspot_com.Echo is called"   
  producer_project_id: "<project>"   
  request_latency_in_ms: 246   
  request_size_in_bytes: 2072   
  response_size_in_bytes: 183   
  service_agent: "ESPv2/2.10.0"   
  service_config_id: "2020-05-29r2"   
  timestamp: 1590794126.0151885   
  url: "/echo"   
 }

I'm not sure what's going on here or if this is an unsupported pattern of deployment, and ESP v2 cannot work in App Engine and has to be Cloud Run.

[maroux]

Almost looks like x-google-backend is not being respected?

[maroux]

Update: seems like this was caused by IPv6. Adding --backend_dns_lookup_family=v4only to ESPv2_ARGS fixes the problem but I don't know if that's expected usage.

[qiwzhang]

I think you deploy ESPv2 in Cloud Run and your backend is deployed in AppEng Flex, is that correct?

If --backend_dns_lookup_family=v4only fixes the problem, it means the DNS name to your AppEng https://-dot-.appspot.com could not be resolved in Ipv6. If that is the case, you can keep use ipv4. I an not sure why ipv6 doesn't work. it works for the DNS name from Cloud Run and Cloud function, and AppEng standard.

[maroux]

No, I'm deploying ESPv2 in App Engine Flex. I don't know why .appspot.com urls fail to resolve with ipv6 either..

[qiwzhang]

How did you deploy ESPv2 in AppEngine Flex? Here is our doc https://cloud.google.com/endpoints/docs/openapi/get-started-app-engine on doing that by specifying endpoints_api_service: in app.yaml file. For such deployment, it doesn't support ESPv2.

-Wayne

On Sat, May 30, 2020 at 3:17 PM Aniruddha Maru [email protected] wrote:

No, I'm deploying ESPv2 in App Engine Flex. I don't know why .appspot.com urls fail to resolve with ipv6 either..

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/esp-v2/issues/170#issuecomment-636392164, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZ6TB7IHS5V56T4Y36BHILRUGAYNANCNFSM4NONMA6Q .

[maroux]

Well, I just deployed ESP docker container to app engine flex using custom runtime (didn't use endpoints_api_service since that doesn't support v2 as you mention). It.. almost works? #171 is still a blocker for me..

[qiwzhang]

I see. I replied in the issue #171 https://github.com/GoogleCloudPlatform/esp-v2/issues/171

On Mon, Jun 1, 2020 at 10:10 AM Aniruddha Maru [email protected] wrote:

Well, I just deployed ESP docker container to app engine flex using custom runtime. It.. almost works? #171 https://github.com/GoogleCloudPlatform/esp-v2/issues/171 is still a blocker for me..

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/esp-v2/issues/170#issuecomment-636992260, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZ6TB4PV7LUEEBS7OXEEQDRUPOHRANCNFSM4NONMA6Q .