Unable to access flatcar-linux.net due to certificate / server errors

[geckolinux]

Hi there, I can't seem to access flatcar-linux.net at all on my desktop or from my cloud VM due to certificate problems, is there something I'm not getting?

Downloading the signature for https://stable.release.flatcar-linux.net/amd64-usr/3227.2.2/flatcar_production_image.bin.bz2...
+ wget --tries 10 --timeout=20 --retry-connrefused --no-verbose -O /tmp/flatcar-install.EaP4jpffkk/flatcar_production_image.bin.bz2.sig https://stable.release.flatcar-linux.net/amd64-usr/3227.2.2/flatcar_production_image.bin.bz2.sig
Read error (The request is invalid.) in headers.

https://flatcar-linux.org/docs/latest/migrating-from-coreos/update-from-container-linux/

If it fails due to SSL connection issues from outdated certificates, you can also download the update payload of the latest Stable release through plain HTTP and use the flatcar-update script instead:

wget --no-check-certificate http://update.release.flatcar-linux.net/amd64-usr/3227.2.2/flatcar_production_update.gz
--2022-10-09 03:37:23--  http://update.release.flatcar-linux.net/amd64-usr/3227.2.2/flatcar_production_update.gz
Resolving update.release.flatcar-linux.net (update.release.flatcar-linux.net)... 136.144.58.113
Connecting to update.release.flatcar-linux.net (update.release.flatcar-linux.net)|136.144.58.113|:80... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

[Jeremy-Boyle]

Also getting issues

wget https://beta.release.flatcar-linux.net/amd64-usr/3346.1.0/flatcar_production_iso_image.iso
--2022-10-08 22:08:27--  https://beta.release.flatcar-linux.net/amd64-usr/3346.1.0/flatcar_production_iso_image.iso
Resolving beta.release.flatcar-linux.net (beta.release.flatcar-linux.net)... 136.144.58.113
Connecting to beta.release.flatcar-linux.net (beta.release.flatcar-linux.net)|136.144.58.113|:443... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2022-10-08 22:08:28--  (try: 2)  https://beta.release.flatcar-linux.net/amd64-usr/3346.1.0/flatcar_production_iso_image.iso
Connecting to beta.release.flatcar-linux.net (beta.release.flatcar-linux.net)|136.144.58.113|:443... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2022-10-08 22:08:31--  (try: 3)  https://beta.release.flatcar-linux.net/amd64-usr/3346.1.0/flatcar_production_iso_image.iso
Connecting to beta.release.flatcar-linux.net (beta.release.flatcar-linux.net)|136.144.58.113|:443... connected.
HTTP request sent, awaiting response... No data received.

[Jeremy-Boyle]

Adding chat from slack https://kubernetes.slack.com/archives/C03GQ8B5XNJ/p1665320134211289

Seems the issue is: I was able to connect via pulling the ip address from outside the united states and using that ip to then curl.

curl -H "beta.release.flatcar-linux.net" https://147.75.87.17 -kvv
*   Trying 147.75.87.17:443...
* Connected to 147.75.87.17 (147.75.87.17) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=stable.release.flatcar-linux.net
*  start date: Sep 29 14:01:41 2022 GMT
*  expire date: Dec 28 14:01:40 2022 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fee6a00ca00)
> GET / HTTP/2
> Host: 147.75.87.17
> user-agent: curl/7.79.1
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx/1.23.1
< date: Sun, 09 Oct 2022 13:35:20 GMT
< content-type: text/html; charset=utf-8
<

Looks like your issue has to do with something happening on 136.144.58.113

;; ANSWER SECTION:
beta.release.flatcar-linux.net.	120 IN	CNAME	update.release.flatcar-linux.net.
update.release.flatcar-linux.net. 120 IN A	136.144.58.113

curl https://beta.release.flatcar-linux.net -vv
*   Trying 136.144.58.113:443...
* Connected to beta.release.flatcar-linux.net (136.144.58.113) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=stable.release.flatcar-linux.net
*  start date: Sep 29 14:01:41 2022 GMT
*  expire date: Dec 28 14:01:40 2022 GMT
*  subjectAltName: host "beta.release.flatcar-linux.net" matched cert's "beta.release.flatcar-linux.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb00c010a00)
> GET / HTTP/2
> Host: beta.release.flatcar-linux.net
> user-agent: curl/7.79.1
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx/1.23.1
< date: Sun, 09 Oct 2022 13:37:52 GMT
< content-type: text/html; charset=utf-8
< 
* HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)
* stopped the pause stream!
* Connection #0 to host beta.release.flatcar-linux.net left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)```

@geckolinux

Try adding 147.75.87.17 beta.release.flatcar-linux.net to your /etc/hosts file , it worked for me for the short time.

[till]

Which DNS servers are you using yourself? I've been pulling images all weekend, no issues.

[Jeremy-Boyle]

It seems like the issue is geo filtering from the United States it works via vpn.

I'm using 1.1.1.1

[till]

Odd indeed, just adding this here — as it seems to be working from "here" when I query with Cloudflare:

dig @1.1.1.1 +short beta.release.flatcar-linux.net
update.release.flatcar-linux.net.
147.75.87.17

Are you by chance in Seattle (or in the vicinity)? Cloudflare seems to have issues locally per their status page. Maybe try 9.9.9.9 instead as a resolver?

[Jeremy-Boyle]

Im actually in Texas.

Ive spun up instances in the following aws regions to test aswell

US West (Oregon) Region US West (Northern California) Region US East (Ohio) Region US East (Northern Virginia) Region

Also tried the following vpn locations that did not work

West Coast Central East Coast

They resolve to 136.144.58.113

Also tried the following DNS servers:

8.8.8.8
8.8.4.4
1.1.1.1
1.0.0.1
9.9.9.9 

AWS Locations outside of the US worked and resolve to 147.75.87.17 Europe (London) Region

VPN worked

Germany London

[geckolinux]

I ran into this problem on a Hetzner Cloud VM, I think it's using their internal DNS (185.12.64.2). Also my workstation is unable to access it using 8.8.8.8 DNS.

[pothos]

Affected is 136.144.58.113 - I've restarted nginx there and it helped but it gets flooded with many requests and it could happen again - we need to tweak the config or find the bug that causes the empty responses. FYI, you can test this with curl --resolve stable.release.flatcar-linux.net:443:136.144.58.113 https://stable.release.flatcar-linux.net/amd64-usr/2079.3.2/version.txt

[Jeremy-Boyle]

Confirmed fixed (for now)!

Also, I couldn't help notice that your nginx is replying with your nginx version when I was curling.

You may want to consider removing the severname from the config or changing it to something that doesn't give the direct version away. Cheers !

[t-lo]

Closing this as we deployed a mitigation which appears to work. Thank you again for the report!

Please re-open if the issue persists / comes back for you.