Pull BGP peering info out of metadata for equinixMetal platform

[rsmitty]

We may be missing some other metadata for the EM platform. A user has reported they need a config patch like:

machine:
  network:
    interfaces:
      - interface: bond0
        routes:
          - network: 169.254.255.1/32
            gateway: 10.66.160.128
          - network: 169.254.255.2/32
            gateway: 10.66.160.128

in order to get BGP peering to work for their nodes there. Looking at https://deploy.equinix.com/developers/docs/metal/bgp/bgp-on-equinix-metal/ it seems that this info could be pulled out of the metadata server as part of the platform setup. We should explore this and include these routes by default if they are present.

[smira]

We need an exact example of what we should translate from the metadata to the machine network config.

The page above only talks about announcing the IP on lo, everything else is about BGP config.

[rsmitty]

Yes agreed. We'll need to go through the process of setting this up in our EM account to see exactly what's needed. But my first assumption is that we just need to add routes to the bond if we have the peer_ips in the BGP neighbors section of the metadata. But yes, testing required here.

[hh]

This is the approach I'm taking for now: https://github.com/ii/infra/commit/a8f05e2a83b90fc82e4be3c073bd1223b91f1c32

It's a first stab, but it does work! Feedback welcome!

[hh]

Here are the resulting routes on bond0:

talosctl  --nodes 139.178.70.71,139.178.70.63,139.178.70.169 get routes | grep bond0 | grep inet4 | grep -v 10.67
139.178.70.71    network     RouteStatus   inet4//139.178.70.70/31/0                                   2         139.178.70.70/31                                           bond0          0
139.178.70.71    network     RouteStatus   inet4/139.178.70.70//1024                                   1                                         139.178.70.70              bond0          1024
139.178.70.71    network     RouteStatus   local/inet4//139.178.70.71/32/0                             114       139.178.70.71/32                                           bond0          0
139.178.70.71    network     RouteStatus   local/inet4//139.178.94.44/32/0                             86        139.178.94.44/32                                           bond0          0
139.178.70.63    network     RouteStatus   inet4//139.178.70.62/31/0                                   4         139.178.70.62/31                                           bond0          0
139.178.70.63    network     RouteStatus   inet4/139.178.70.62//1024                                   1                                         139.178.70.62              bond0          1024
139.178.70.63    network     RouteStatus   local/inet4//139.178.70.63/32/0                             129       139.178.70.63/32                                           bond0          0
139.178.70.169   network     RouteStatus   inet4//139.178.70.168/31/0                                  4         139.178.70.168/31                                          bond0          0
139.178.70.169   network     RouteStatus   inet4/139.178.70.168//1024                                  1                                         139.178.70.168             bond0          1024
139.178.70.169   network     RouteStatus   local/inet4//139.178.70.169/32/0                            130       139.178.70.169/32                                          bond0          0

[smira]

@rsmitty is this what you've been looking for? routes to the BGP peers?

(I don't think we should announce any IPs)

[hh]

Pull from here : https://deploy.equinix.com/developers/docs/metal/server-metadata/metadata/#bgp-neighbors

I'd say this would be a reasonable addition to talos.platform=equinixMetal automation and populate the networking fully:

https://github.com/siderolabs/talos/blob/main/internal/app/machined/pkg/runtime/v1alpha1/platform/equinixmetal/equinix.go#L251