chrome-for-testing
chrome-for-testing copied to clipboard
GoogleChromeLabs
edgedl.me.gvt1.com is not working properly as of 7:40am GMT
Hello,
Not sure if this is the right place but it's the place that is Chrome For Testing.
We noticed this morning that downloads from edgedl.me.gvt1.com are no longer working on a range of EC2 servers this morning. We have hundreds of servers, all configured identical and about half of them cannot connect to edgedl.me.gvt1.com
curl -v https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/117.0.5938.149/linux64/chrome-linux64.zip
* Trying 34.104.35.123:443...
* TCP_NODELAY set
* Trying 2600:1900:4110:86f:::443...
* TCP_NODELAY set
* Immediate connect fail for 2600:1900:4110:86f::: Cannot assign requested address
* TCP_NODELAY set
* Trying 2600:1900:4110:86f:::443...
* TCP_NODELAY set
* Immediate connect fail for 2600:1900:4110:86f::: Network is unreachable
* Trying 2600:1900:4110:86f:::443...
* TCP_NODELAY set
* Immediate connect fail for 2600:1900:4110:86f::: Network is unreachable
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0* Trying 2600:1900:4110:86f:::443...
* TCP_NODELAY set
Has something changed with edgedl.me.gvt1.com ? or are there any mirrors?
In our company we have same problems. We can reach https://edgedl.me.gvt1.com from developers machine but we cannot reach it from CI runners. It looks like we were block by some kind of firewall rules on server side
In our company we have same problems. We can reach
https://edgedl.me.gvt1.comfrom developers machine but we cannot reach it from CI runners. It looks like we were block by some kind of firewall rules on server side
this is similar. Infact even some of our AWS EC2 servers can connect. Server A will connect, Server B cannot connect. Both are 100% open to the public on AWS Firewalls, both have identical Network and VPC setups. It's very strange...
We have had to move all dockers that use this to another EC2 cluster which seems to ping, but this has only broken this morning... It's been fine for.. a very long time!
@vekien are there any ongoing AWS incidents? we are also able to connect from our networks.
It seems to be ipv6 issue, https://github.com/puppeteer/puppeteer/issues/11060#issuecomment-1745010716
With curl use --ipv4 flag and with node / npm --dns-result-order=ipv4first
It seems to be ipv6 issue, puppeteer/puppeteer#11060 (comment)
With curl use
--ipv4flag and with node / npm--dns-result-order=ipv4first
We had issues even forcing ipv4 as well as disabling ipv6 completely.
Wonder if it's slightly separate issue or something along the routing with that url.
@vekien are there any ongoing AWS incidents? we are also able to connect from our networks.
No incident but from experience AWS rarely tell us.
In typical fashion after moving all the dockers to working servers it seems to be now working in all servers where it was previously failing, so it seemed to be short lived issue...
I noticed curl tests went from: Network is unreachable --> Cannot assign requested address --> Connected to edgedl.me.gvt1.com (34.104.35.123) port 443 (#0)
So it now works fine! I will keep monitoring over the day, maybe it was some routing issue on AWS end.
curl -v -O https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/117.0.5938.149/linux64/chrome-linux64.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 34.104.35.123:443...
* Connected to edgedl.me.gvt1.com (34.104.35.123) port 443 (#0)
* ALPN: offers h2,http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
* CApath: none
.....
Since Chrome dev downloads moved to edgedl.me.gvt1.com, there have been constant connection problems. In most cases the problem seems to be that TLS negotiation fails after "Client Hello". From the downloader perspective, it seems that edgedl service has backends that behave differently than others and when connection initiation starts on a problematic service node it fails. Different connection issues seem to be commented on many issues regarding Chrome dev tool downloads after service changes, but the issue hasn't reached edgedl.me.gvt1.com service admins. Retries are required for the download to be successful.
~ Fri Oct 6 09:33:50 EEST 2023
curl --trace-time --trace-ascii - -4 -Io /dev/null https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/117.0.5938.92/linux64/chromedriver-linux64.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 009:33:52.741048 == Info: Trying 34.104.35.123:443...
09:33:52.766018 == Info: Connected to edgedl.me.gvt1.com (34.104.35.123) port 443 (#0)
09:33:52.766120 == Info: ALPN: offers h2,http/1.1
09:33:52.766313 == Info: (304) (OUT), TLS handshake, Client hello (1):
09:33:52.766318 => Send SSL data, 323 bytes (0x143)
0000: ...?..5.._.H.s...>.w_..2...DE.-....... x..`......&..G.:...4.Jf..
0040: {V=e..].b.............0.,.(.$.......k.9...........=.5...../.+.'.
0080: #.......g.3...E...<./...A.......................+............3.&
00c0: .$... ..^..7..........CI.....|.....E.W.........edgedl.me.gvt1.co
0100: m.......................................................h2.http/
0140: 1.1
09:33:52.773021 == Info: CAfile: /etc/ssl/cert.pem
09:33:52.773039 == Info: CApath: none
0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 009:34:09.369026 == Info: LibreSSL/3.3.6: error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure
0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0
09:34:09.369364 == Info: Closing connection 0
curl: (35) LibreSSL/3.3.6: error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure
curl 8.1.2 (x86_64-apple-darwin23.0) libcurl/8.1.2 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.55.1
~ Fri Oct 6 09:32:26 EEST 2023
curl --trace-ascii - -4 -Io /dev/null https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/117.0.5938.92/linux64/chromedriver-linux64.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0== Info: Trying 34.104.35.123...
== Info: TCP_NODELAY set
== Info: Connected to edgedl.me.gvt1.com (34.104.35.123) port 443 (#0)
== Info: ALPN, offering h2
== Info: ALPN, offering http/1.1
== Info: successfully set certificate verify locations:
== Info: CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
=> Send SSL data, 5 bytes (0x5)
0000: .....
== Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
=> Send SSL data, 512 bytes (0x200)
0000: ......{..H..m.....{"Z%P...ZM...q.9E[. .D.A..9.s..:.s.&.R......
0040: 3F?.i...H.........,.0.......+./...#.'.................=.<.5./...
0080: ........k.g.9.3.....k.........edgedl.me.gvt1.com................
00c0: ........3t.........h2.http/1.1.........1.....&.$................
0100: .....................+........-.....3.&.$... %.yF...2...,5 .D.R.
0140: .[..;@?...[[s...................................................
0180: ................................................................
01c0: ................................................................
0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0<= Recv SSL data, 5 bytes (0x5)
0000: .....
== Info: TLSv1.3 (IN), TLS alert, handshake failure (552):
<= Recv SSL data, 2 bytes (0x2)
0000: .(
== Info: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0
== Info: Closing connection 0
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.2.0) libssh/0.9.6/openssl/zlib nghttp2/1.33.0
Got:
57.65 npm ERR! code: 'ECONNRESET',
57.65 npm ERR! path: null,
57.65 npm ERR! host: 'edgedl.me.gvt1.com',
57.65 npm ERR! port: 443,
57.65 npm ERR! localAddress: undefined
For VIVO ISP at south Brazil, this is weird, when connected to a USA VPN it worked, not blocked anymore.
