envoy icon indicating copy to clipboard operation
envoy copied to clipboard

Mobile Android CI failing with CloudFront 403 from api.lyft.com

Open phlax opened this issue 5 months ago • 3 comments

some tests appear to rely on http://api.lyft.com/ping which can return 403


06-19 10:01:06.540  6115  6149 D MainActivity: [2025-06-19 10:01:06.540][6149][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1895] [Tags: "ConnectionId":"12345","StreamId":"105507803397563138"] encoding headers via codec (end_stream=false):
06-19 10:01:06.540  6115  6149 D MainActivity: ':status', '403'
06-19 10:01:06.540  6115  6149 D MainActivity: 'server', 'CloudFront'
06-19 10:01:06.540  6115  6149 D MainActivity: 'date', 'Thu, 19 Jun 2025 10:01:06 GMT'
06-19 10:01:06.540  6115  6149 D MainActivity: 'content-type', 'text/html'
06-19 10:01:06.540  6115  6149 D MainActivity: 'content-length', '919'
06-19 10:01:06.540  6115  6149 D MainActivity: 'x-cache', 'Error from cloudfront'
06-19 10:01:06.540  6115  6149 D MainActivity: 'via', '1.1 9e9fde687cd38d3d64c729260b6189f6.cloudfront.net (CloudFront)'
06-19 10:01:06.540  6115  6149 D MainActivity: 'x-amz-cf-pop', 'LAX53-P1'
06-19 10:01:06.540  6115  6149 D MainActivity: 'x-amz-cf-id', 'ydLvdBkoXUJXYm6my6eEFXU_66BzrA9rzi0Nu89YXE7XXYJwgFYNMw=='
06-19 10:01:06.540  6115  6149 D MainActivity: 'x-envoy-upstream-service-time', '9'
06-19 10:01:06.540  6115  6149 D MainActivity: 'x-envoy-attempt-count', '1'
06-19 10:01:06.540  6115  6149 D MainActivity: 'async-filter-demo', '1'
06-19 10:01:06.540  6115  6149 D MainActivity: 'buffer-filter-demo', '1'
06-19 10:01:06.540  6115  6149 D MainActivity: 'filter-demo', '1'
06-19 10:01:06.540  6115  6149 D MainActivity: 

https://github.com/envoyproxy/envoy/actions/runs/15754921487/job/44408290255#step:17:44355

im thinking probs it would be better if this was mocked in some way, so not relying on external endpoint/s etc

phlax avatar Jun 19 '25 10:06 phlax

cc @abeyad

phlax avatar Jun 19 '25 10:06 phlax

I'm surprised this test hasn't failed before with 403? Is it a flake or a consistent failure? I agree mocking would be better. Unfortunately a fair bit of tests use an external endpoint, we need to address that.

abeyad avatar Jun 19 '25 14:06 abeyad

not sure if it fails frequently or not - retest fixed it, so defo transient

phlax avatar Jun 19 '25 15:06 phlax

@abeyad debugging the issues with timeouts when starting/testing ios simulators - this looks like the actual cause

most times it gets back the expected headers almost immediately - but sometimes it times out after 5 minutes - looking through the logs and its making a stack of calls to this api - so its a strong candidate for what could cause such variability

phlax avatar Jun 20 '25 10:06 phlax

looking through the logs and its making a stack of calls to this api

which API are you referring to?

abeyad avatar Jun 23 '25 15:06 abeyad

api.lyft.com

just realized this is for android and not ios - so this isnt actually fixed ..

phlax avatar Jun 23 '25 15:06 phlax

 06-24 15:51:08.108  6099  6128 D MainActivity: 'server', 'CloudFront'
06-24 15:51:08.108  6099  6128 D MainActivity: 'date', 'Tue, 24 Jun 2025 15:51:08 GMT'
06-24 15:51:08.108  6099  6128 D MainActivity: 'content-type', 'text/html'
06-24 15:51:08.108  6099  6128 D MainActivity: 'content-length',` '919'
06-24 15:51:08.108  6099  6128 D MainActivity: 'x-cache', 'Error from cloudfront'
06-24 15:51:08.108  6099  6128 D MainActivity: 'via', '1.1 a19e5a78228f461a542b5c5f6d13243e.cloudfront.net (CloudFront)'
06-24 15:51:08.108  6099  6128 D MainActivity: 'x-amz-cf-pop', 'PHX52-P1'
06-24 15:51:08.108  6099  6128 D MainActivity: 'x-amz-cf-id', '8iOXSrg_qYkvZxpOOoWkqPkIXe3wFhkBm50_QSqtTrfgM5EieoU9vA=='
06-24 15:51:08.108  6099  6128 D MainActivity: 'x-envoy-upstream-service-time', '3'
06-24 15:51:08.108  6099  6128 D MainActivity: 'x-envoy-attempt-count', '1'
06-24 15:51:08.108  6099  6128 D MainActivity: 

phlax avatar Jun 24 '25 18:06 phlax

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions[bot] avatar Jul 24 '25 20:07 github-actions[bot]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

github-actions[bot] avatar Jul 31 '25 20:07 github-actions[bot]