Scale node storage based on pod ephemeral-storage requests

[dewjam]

Tell us about your request What do you want us to build? Enable Karpenter to dynamically scale the size of block device attached to a node at provision time. The size of blocked device would be based on the sum of ephemeral-storage requests of pods being bin-packed to a node plus some overhead.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Currently, a node's storage capacity can be defined through the Karpenter provisioner through Block Device Mappings. This works well, but forces customers to define a static value for all instances launched through a given provisioner. Customers would like the ability to dynamically scale storage of the nodes based on the pod workload or by instance-type.

Are you currently working around this issue? This can be worked around by defining Block Device Mappings in the Karpenter Provisioner. These values are static for the given provisioner, however, and cannot be dynamically scaled up/down.

Related issues: #2512 #2298 #1995 #1467 #3077 #3111

Additional context Anything else we should know?

Attachments If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[cx-IliyaY]

is there some progress with it? because the only answer is this link but it's dead now error 404... Block Device Mappings

[tzneal]

Those docs are now here, but there hasn't been any update.

[jonathan-innis]

is there some progress with it?

No current, active progress. But it's something that's considered "v1" scope, which means that we're planning to work on this part of the v1 release for Karpenter. It's definitely within our list of priorities but the maintainer team has been somewhat time-constrained lately and working on some other feature work and stability improvements.

[jagadeesh-kancherla-tfs]

+1

[pragmaticivan]

This would reduce some alerting for NodePools sharing the same StorageClass with static storage values.

2xl - 16xl might require a bit of a drift in storage due to multiple factors, including docker pulls and volumes.

Any chance this would get a bump in priority?

[Smana]

Are there any update on this issue? We currently need to specify a constraint for using nvme instance types. And prepare the RAID0 array.

        - key: karpenter.k8s.aws/instance-local-nvme
          operator: Gt
          values: ["100"]

Otherwise we face issues for some pods not having enough ephemeral disk space which leads to evicted pods with a node DiskPressure error.

[zakariais]

is there some progress with it?

[MedAzizTousli]

Any progress on this?

[pkit]

@Smana

       - key: karpenter.k8s.aws/instance-local-nvme
          operator: Gt
          values: ["100"]

Doesn't seem to work for me at all, all nodes have hardcoded 20GB ephemeral no matter what.

[liorfr-monday]

+1

[hari2192]

What is the update on this ?

We are running karpenter on cluster where workloads requests are mainly factored on two fronts - Memory and Ephemeral requests and karpenter not having capability to consider ephemeral requests , actually raising questions on decision to launch the most efficient node for cluster.

[bittracer]

What is the update on this ?

We are running karpenter on cluster where workloads requests are mainly factored on two fronts - Memory and Ephemeral requests and karpenter not having capability to consider ephemeral requests , actually raising questions on decision to launch the most efficient node for cluster.

I’m in a similar situation and would love to get some insights on the tentative planned release date for this feature. Any updates would be greatly appreciated!

[blakeromano]

I saw https://github.com/aws/karpenter-provider-aws/issues/3111 attached to this but it seems like the requests are slightly different (but could be implemented similarly)

The use case we have is that we are using Karpenter with containers with variable size and we are seeing that nodes can get pods into a stuck state based on missing disk space so we want to be able to say for each cpu give 4x the amount to the root disk for image space.

[Usman2907]

Confirming this issue persists in Kubernetes v1.32 with Karpenter 1.5.0. Despite environment/configuration changes, ephemeral storage remains capped at 20Gi. When scheduling pods requiring significant ephemeral storage, they consistently fail due to disk pressure errors after sustained runtime. This occurs even when instance types theoretically support larger storage.

[theadzik]

We came into this problem because logs were missing from Completed and Errored pods. Turned out it was kubelet doing garbage collection and deleting containers because the puny 20GB disk got full.