eksctl icon indicating copy to clipboard operation
eksctl copied to clipboard

[Feature] Support KubeletExtraConfig for Windows nodegroups

Open cthrasher opened this issue 2 years ago • 0 comments

When trying to create a Windows nodegroup, I got the error:

Error: couldn't create cluster provider from options: kubeletExtraConfig is not supported for WindowsServer2019FullContainer nodegroups (path=nodeGroups[0].kubeletExtraConfig)

However, the aws-eks-best-practices documentation specifically calls out the need to do this to get around the lack of an out-of-memory process killer in Windows itself:

If eksctl is the deployment tool, check the following documentation to customize the kubelet configuration https://eksctl.io/usage/customizing-the-kubelet/

The CloudFormation template referenced in that documentation takes the kubelet arguments as a parameter named BootstrapArguments and injects them into a custom bootstrap script written in Powershell:

<powershell>
[string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
[string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
[string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
[string]$cfn_signal = "$env:ProgramFiles\Amazon\cfn-bootstrap\cfn-signal.exe"
& $EKSBootstrapScriptFile -EKSClusterName ${ClusterName} ${BootstrapArguments} 3>&1 4>&1 5>&1 6>&1
$LastError = if ($?) { 0 } else { $Error[0].Exception.HResult }
& $cfn_signal --exit-code=$LastError --stack="${AWS::StackName}" --resource="NodeGroup" --region=${AWS::Region}
</powershell>

In order to set up the nodegroup with these memory settings, we have to be able to specify kubeletExtraConfig with Windows nodes. The specific arguments they call out would look like this:

    kubeletExtraConfig:
      kubeReserved:
        memory: "0.5Gi"
        ephemeral-storage: "1Gi"
      systemReserved:
        memory: "1.5Gi"
        ephemeral-storage: "1Gi"
      evictionHard:
        memory.available: "200Mi"
        nodefs.available: "10%"

cthrasher avatar Aug 25 '22 20:08 cthrasher