IPv6-Only Cluster Support

[soriaj]

Description

Installation of IPv6 only kURL cluster requires modifying the ClusterConfiguration and InitConfiguration during installation.

Reproduction Steps

1. Create a new Kubernetes (kURL) installer (spec below): NOTE: Combination of add-ons that fail to deploy or function with IPv6 (see Additional Information)

apiVersion: cluster.kurl.sh/v1beta1
kind: Installer
metadata:
  name: ""
spec:
  containerd:
    version: latest
  kubernetes:
    version: 1.21.5
  antrea:
    version: latest
  contour:
    version: latest
  prometheus:
    version: latest
  openebs:
    version: "2.6.0"
    namespace: "openebs"
    isLocalPVEnabled: true
    localPVStorageClassName: "default"
  kotsadm:
    version: latest
    disableS3: true

2. Copy the kURL generated installer script to your system.
3. Modify the installer script as described below prior to deployment. NOTE: Air Gap - transfer kURL air gap bundle to instance and un-package tar file

• Search for kubeadm init \ that is uncommented
• Add a pause function (e.g pause 'Press [Enter] key to continue...)'
• Above function init() add the below pause function

function pause(){
   read -p "$*"
}

4. Start the install script sudo bash {Your_installer_script}.sh and wait for pause
5. Copy the generated kubeadm.conf file to modify.

cp /opt/replicated/kubeadm.conf kubeadm.conf

6. IPv6 Node address, CIDR(s) and controllerManager extraArgs need to be added. Example below:

apiServer:
  certSANs:
  - fde4:8dba:82e1:98d::c4 # Node IPv6 Address
  extraArgs:
    service-node-port-range: 80-60000
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: # Needed for Antrea CNI
  extraArgs:
    allocate-node-cidrs: "true"
    cluster-cidr: fde4:8dba:82e1:98d:8:2::/56
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
kind: ClusterConfiguration
kubernetesVersion: v1.21.5
networking:
  serviceSubnet: fde4:8dba:82e1:98d:8:3::/112 # IPv6 Service CIDR
  podSubnet: fde4:8dba:82e1:98d:8:2::/64 #  IPv6 Pod CIDR
useHyperKubeImage: false
---
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: fqbrck.5bdzy8ebfio6lrgz
  ttl: null
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress:  fde4:8dba:82e1:98d::c4 # Node IPv6 Address
nodeRegistration:
  kubeletExtraArgs:
    container-runtime: remote
    container-runtime-endpoint: unix:///run/containerd/containerd.sock
    hostname-override: ip-10-0-1-211 # Node hostname
    node-ip: fde4:8dba:82e1:98d::c4 # Node IPv6 Address
    node-labels: kurl.sh/cluster=true
  name: ip-10-0-1-211 # Node hostname
  taints: []
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
---

7. Replace existing file

sudo cp kubeadm.conf /opt/replicated/kubeadm.conf

8. Return to paused script and continue install
9. Once complete you will have a running IPv6 cluster

NAME               STATUS   ROLES                  AGE   VERSION   INTERNAL-IP                            EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
ip-172-24-103-73   Ready    control-plane,master   66s   v1.21.5   2600:1f1c:7:611a:f34f:8484:3255:5650   <none>        Ubuntu 20.04.3 LTS   5.11.0-1020-aws   containerd://1.4.6

Expected Behavior

IPv6 support should be provided without manual modification of ClusterConfiguration and InitConfiguration during installation.

Additional Information

Node Spec:

Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

CPU: 4
Memory:16GB
SSD:200GB

Non running IPv6 kURL spec(s) with add-ons

Halts on minio

apiVersion: cluster.kurl.sh/v1beta1
kind: Installer
metadata:
  name: ""
spec:
  containerd:
    version: latest
  kotsadm:
    version: latest
  kubernetes:
    version: 1.21.5
  longhorn:
    version: latest
  antrea:
    version: latest
  minio:
    version: latest

Halts on kotsadm

apiVersion: cluster.kurl.sh/v1beta1
kind: Installer
metadata:
  name: ""
spec:
  containerd:
    version: latest
  kotsadm:
    version: latest
    disableS3: true
  kubernetes:
    version: 1.21.5
  antrea:
    version: latest
  longhorn:
    version: latest

Halts on registry

apiVersion: cluster.kurl.sh/v1beta1
kind: Installer
metadata:
  name: ""
spec:
  containerd:
    version: latest
  kubernetes:
    version: 1.21.5
  antrea:
    version: latest
  contour:
    version: latest
  prometheus:
    version: latest
  openebs:
    version: "2.6.0"
    namespace: "openebs"
    isLocalPVEnabled: true
    localPVStorageClassName: "default"
  kotsadm:
    version: latest
    disableS3: true
  registry:
    version: latest

[marccampbell]

@areed do you want to take a look at this?