Support configuration to disable DHCP DNS servers

[adobley]

trafficstars

/kind feature

Describe the solution you'd like As a user, I would like to toggle a setting (ignoreDHCPNameservers) that will cause nodes to ignore the nameservers that DHCP assigns, so that I have full control over my DNS servers for my cluster.

This can be done with netplan already through the dhcp{4,6}-overrides.use-dns configuration option. However this currently only has an effect on the networkd backend. netplan docs

We would like to add a field to the NetworkDeviceSpec on the VsphereMachineTemplate, something like:

// IgnoreDHCPNameservers is a flag that indicates whether or not to use the
// DNS servers received from the DHCP server.
// If true the DNS servers from the DHCP server will be ignored and only
// the statically configured nameservers will be added to the devices DNS
// servers.
// If false the DNS servers from DHCP and any statically configured
// nameservers will be added to the devices DNS servers.
// Please note that this only works with the `networkd` backend for netplan.
// +optional
IgnoreDHCPNameservers bool `json:"ignoreDHCPNameservers,omitempty"`

An example simplified config without all fields

---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate
metadata:
  name: my-cluster
  namespace: my-namespace
spec:
  template:
    spec:
      network:
        devices:
          - networkName: "my-favorite-network"
            dhcp4: true
            dhcp6: true
            nameservers: [8.8.8.8, 8.8.4.4, 2001:4860:4860::8888]
            ignoreDHCPNameservers: true

An example metadata output

instance-id: "test-vm"
local-hostname: "test-vm"
wait-on-network:
  ipv4: true
  ipv6: true
network:
  version: 2
  ethernets:
    id0:
      match:
        macaddress: "00:00:00:00:00"
      set-name: "ens192"
      wakeonlan: true
      dhcp4: true
      dhcp6: true
      dhcp4-overrides:
        use-dns: false
      dhcp6-overrides:
        use-dns: false
      nameservers:
        addresses:
        - 8.8.8.8
        - 8.8.4.4
        - 2001:4860:4860::8888

The resulting netplan yaml

network:
  ethernets:
    id0:
      dhcp4: true
      dhcp4-overrides:
        use-dns: false
      dhcp6: true
      dhcp6-overrides:
        use-dns: false
      match:
        macaddress: xx:xx:xx:xx:xx:xx
      nameservers:
        addresses:
        - 8.8.8.8
        - 8.8.4.4
        - 2001:4860:4860::8888
      set-name: eth0
      wakeonlan: true
  version: 2

We'd love to get any input on the design and resolve any concerns there might be.

Anything else you would like to add: This is related to https://github.com/vmware-tanzu/tanzu-framework/issues/1103

We are happy to PR this change and we have one started so we could explore our design. We are struggling a bit with running the e2e tests on our own and could use any help there as well but that might be something better to drop into k8s slack.