Can't figure out how to change the default instance type m5.large

[NinoSkopac]

Describe the documentation issue

I'd like to be able to set the default instance type that gets created with the cluster. It's m5.large. I'm using Karpenter for autoscaling and works great. Karpenter eventually stops working if I simply set the asg desired capacity to 0 because it doesn't have infra (the default m5.large) to run on to create new nodes.

The code in the docs is outdated, I believe, because it's causing TS errors.

const props: AsgClusterProviderProps = {
    minSize: 1,
    maxSize: 10,
    desiredSize: 4,
    version: "auto",
    instanceType: new InstanceType('m5.large'),
    machineImageType: eks.MachineImageType.AMAZON_LINUX_2,
    updatePolicy: UpdatePolicy.Rolling
}
const clusterProvider = new blueprints.AsgClusterProvider(props);
new blueprints.EksBlueprint(scope, { id: 'blueprint', [], [], clusterProvider });

Links

https://aws-quickstart.github.io/cdk-eks-blueprints/cluster-providers/asg-cluster-provider/

[shapirov103]

@NinoSkopac if there is no capacity to run the autoscaler (Karpenter) then you won't get autoscaling - this works as expected. I observe some folks using the pattern of running Karpenter on Fargate without any node groups and relying on Karpenter provisioner to choose the right instance. Instance types can be specified in the requirements attribute of the addon. For example:

 [
   { 
      key: "karpenter.k8s.aws/instance-category",
      operator: "In",
      values: ["c", "m", "r"]
   },
   {
     key: "karpenter.k8s.aws/instance-cpu",
     operator: 'In', 
     values: ["4", "8", "16", "32"]
  }
]

You can find more info on Karpenter provisioner in the docs.

Please let me know if the above addressed your issue.

[NinoSkopac]

@shapirov103 We're on the same page in regards to functionality. My question is about the default m5.large instance type that ships with eks blueprints. In other words, it's not about instances that Karpenter creates.

How to customize the default m5.large instance?

For example:

• change instance type
• don't run workloads on it
• set its own ttlSecondsUntilExpired value, like I can set with Karpenter for nodes it manages

[shapirov103]

Please consider using FargateClusterProvider - this won't create any node groups. Please see example of using FargateClusterProvider here. You can then deploy Karpenter and since Fargate is using node per pod structure nothing else will be sharing a node with Karpenter.

However, to answer your question, you can check this blog post section on setting the default instance type.

To prevent workloads running on a node group you can add a NoSchedule or PreferNoSchedulealong with CriticalAddonsOnly taint to nodes in the node group (using taints attribute on MngClusterProviderProps). This will prevent new pods from being scheduled on the nodes, while allowing existing pods to remain. You can then use tolerations to schedule Karpenter onto these nodes (by default will use CriticalAddonsOnly toleration).

ttlSecondsUntilExpired is an attribute specific to the Karpenter provisioner and is not supported with node groups.

[NinoSkopac]

Thanks for this amazing answer @shapirov103

Right now, I'm facing an issue.

So, I'm following this method you mentioned, and my code is like this:

my-blueprints.ts offending excerpt

import {aws_ec2, aws_iam} from "aws-cdk-lib";
import {MngClusterProvider} from "@aws-quickstart/eks-blueprints";

const managedClusterProvider = new MngClusterProvider({
    minSize: 1,
    maxSize: 2,
    desiredSize: 1,
    instanceTypes: [
        new aws_ec2.InstanceType('c7g.2xlarge'),
    ],
})

And the issue is:

S2322: Type
import("/Users/onin/dev/demo-app/my-blueprints/node_modules/aws-cdk-lib/aws-ec2/lib/instance-types").InstanceType
is not assignable to type
import("/Users/onin/dev/demo-app/my-blueprints/node_modules/@aws-quickstart/eks-blueprints/node_modules/aws-cdk-lib/aws-ec2/lib/instance-types").InstanceType
Types have separate declarations of a private property instanceTypeIdentifier

my-blueprints/package.json

{
  "name": "my-blueprints",
  "version": "0.1.0",
  "bin": {
    "my-blueprints": "bin/my-blueprints.js"
  },
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "test": "jest",
    "cdk": "cdk"
  },
  "devDependencies": {
    "@types/jest": "^29.5.1",
    "@types/node": "20.1.7",
    "jest": "^29.5.0",
    "ts-jest": "^29.1.0",
    "ts-node": "^10.9.1",
    "typescript": "~5.1.3"
  },
  "dependencies": {
    "@aws-quickstart/eks-blueprints": "^1.13.1",
    "constructs": "^10.0.0",
    "source-map-support": "^0.5.21"
  }
}

[shapirov103]

The error is indicative of a version mismatch between CDK that you have installed globally and the version used by the blueprints. Blueprints has a dependency on CDK 2.115.0, see instructions here.

The most straightforward solution is for you to install CDK 2.115.0 globally, then run:

rm -rf node_modules
rm package-lock.json
npm i

If you would like to have a different version of the global CDK and only use 2.115 with blueprints, then you should add blueprints CDK dependency to you package.json as shown here.

After that you can run npx cdk list or npx cdk deploy instead of cdk list.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 60 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions[bot]]

Issue closed due to inactivity.