Unstable get cloud metadata from "magic" IP

[lmq1999]

Description

Unstable to get metadata via curl http://169.254.169.254/openstack

Impact

Without metadata from cloud, I can't run some service and kubernetes CSI

Environment and steps to reproduce

1. Set-up: Flatcar image: flatcar_production_openstack_image.img

pool-g4dzrku5-sj3dtqihuu6cjof6-node-hshs3fbx ~ # cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3975.2.0
VERSION_ID=3975.2.0
BUILD_ID=2024-08-05-2103
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3975.2.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3975.2.0:*:*:*:*:*:*:*"

2. Task: Setup network config for vpn interface and loopback interface

pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # cat /etc/systemd/network
network/       networkd.conf  
pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # cat /etc/systemd/network/
.keep_sys-apps_systemd-0  kengine.network           lo.network                
pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # cat /etc/systemd/network/kengine.network 
[Match]
Name=kengine

[Link]
Unmanaged=yes
pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # cat /etc/systemd/network/lo.network      
[Match]
Name=lo

[Network]
Address=127.0.0.1/8
Address=10.93.0.1/32
pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # 

3. Action(s): Restart networkd and try to get metadata

pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # systemctl restart systemd-networkd
pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # curl http://169.254.169.254/openstack
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27
latestpool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # curl http://169.254.169.254/openstack
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27
latestpool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # curl http://169.254.169.254/openstack
curl: (28) Failed to connect to 169.254.169.254 port 80 after 135318 ms: Couldn't connect to server
pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # curl http://169.254.169.254/openstack
curl: (28) Failed to connect to 169.254.169.254 port 80 after 134717 ms: Couldn't connect to server

After restart networkd again

latestpool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # curl http://169.254.169.254/openstack
curl: (28) Failed to connect to 169.254.169.254 port 80 after 135318 ms: Couldn't connect to server
pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # curl http://169.254.169.254/openstack
curl: (28) Failed to connect to 169.254.169.254 port 80 after 134717 ms: Couldn't connect to server
pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # curl http://169.254.169.254/openstack
curl: (28) Failed to connect to 169.254.169.254 port 80 after 134778 ms: Couldn't connect to server
pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # systemctl restart systemd-networkd
pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # curl http://169.254.169.254/openstack
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27
latestpool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # curl http://169.254.169.254/openstack
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27
latestpool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # curl http://169.254.169.254/openstack
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27

4. Error: curl: (28) Failed to connect to 169.254.169.254 port 80 after 134778 ms: Couldn't connect to server

Expected behavior

Can get metadata like this:

curl http://169.254.169.254/openstack
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27
latest

Additional information

Ip route

pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # ip r
default via 103.107.182.1 dev eth0 proto dhcp src 103.107.182.231 metric 1024 
10.20.4.0/24 dev eth1 proto kernel scope link src 10.20.4.231 metric 1024 
10.20.4.3 dev eth1 proto dhcp scope link src 10.20.4.231 metric 1024 
10.200.9.0/24 via 10.20.4.129 dev eth1 proto kernel 
10.200.16.0/24 via 10.20.4.137 dev eth1 proto kernel 
10.200.71.0/24 via 10.200.71.46 dev cilium_host proto kernel src 10.200.71.46 
10.200.71.46 dev cilium_host proto kernel scope link 
10.200.75.0/24 via 10.20.4.77 dev eth1 proto kernel 
10.200.79.0/24 via 10.20.4.136 dev eth1 proto kernel 
10.200.85.0/24 via 10.20.4.186 dev eth1 proto kernel 
10.200.86.0/24 via 10.20.4.8 dev eth1 proto kernel 
10.200.88.0/24 via 10.20.4.124 dev eth1 proto kernel 
10.200.90.0/24 via 10.20.4.81 dev eth1 proto kernel 
103.107.182.0/24 dev eth0 proto kernel scope link src 103.107.182.231 metric 1024 
103.107.182.1 dev eth0 proto dhcp scope link src 103.107.182.231 metric 1024 
103.107.182.7 dev eth0 proto dhcp scope link src 103.107.182.231 metric 1024 
169.254.169.254 via 103.107.182.7 dev eth0 proto dhcp src 103.107.182.231 metric 1024 
169.254.169.254 via 10.20.4.3 dev eth1 proto dhcp src 10.20.4.231 metric 1024 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 

networkctl list

pool-x45g4eed-x9ipxe2qz7c3offo-node-oes7cdpq /home/core # networkctl list  
IDX LINK        TYPE     OPERATIONAL SETUP     
  1 lo          loopback routable    configured
  2 eth0        ether    routable    configured
  3 eth1        ether    routable    configured
  4 docker0     bridge   no-carrier  unmanaged
  7 cilium_net  ether    degraded    unmanaged
  8 cilium_host ether    routable    unmanaged
 10 lxc_health  ether    degraded    unmanaged

7 links listed.

bonus: I also use docker openvpn config from this issue: https://github.com/flatcar/Flatcar/issues/1515