bottlerocket-os
Bottlerocket 1.48.0 EKS 1.34 image will not mount EBS volumes over nvme15
Image I'm using: EKS 1.34 Bottlerocket AMI 1.48.0 (arm64)
What I expected to happen: All volumes attached to EC2 are properly mounted and available for kubernetes persistent volumes
What actually happened:
- Volumes with an index greater than 15 are not mounted
- Kubernetes pods that require the associated PV will not start
- This issue wasn't present in EKS 1.33 AMIs, and only started after upgrading to EKS 1.34 AMIs.
- Upgrading to Bottlerocket 1.49.0 did not resolve the issue
The EC2 instance shows the EBS volumes attached in the EC2 console, but SSHing into the instance and running lsblk shows that volumes with indexes greater than 15 are not properly mounted.
The following errors occur when trying to mount /dev/xvdao and subsequent devices.
Oct 23 06:32:49 ip-10-128-14-122.ec2.internal kubelet[2201]: E1023 06:32:49.753731 2201 csi_attacher.go:401] kubernetes.io/csi: attacher.MountDevice failed: rpc error: code = NotFound desc = Failed to find device path /dev/xvdap. no device path for device "/dev/xvdap" volume "vol-08078f8696e14b62c" found
Kernel logs showed NVMe device timeout issues for devices after nvme15 (nvme16, nvme17, etc.)
[ 1000.337730] nvme nvme16: I/O tag 25 (1019) QID 1 timeout, completion polled
[ 1030.397337] nvme nvme16: I/O tag 26 (101a) QID 1 timeout, completion polled
[ 1032.307305] nvme nvme17: I/O tag 0 (1000) QID 0 timeout, completion polled
How to reproduce the problem: In an EKS 1.34 cluster, attempt to run more than 15 pods with persistent volumes on a single node
I have opened a case with AWS support regarding this issue but have seen no traction on it, so raising it here for additional awareness. The agent assigned to the case was able to reproduce the issue.
I have worked around the problem by running smaller nodes, so that we don't hit the 15-volume limit because fewer pods are able to schedule on the smaller nodes. However, this is costing us a lot of money as our observability vendor charges us by the node, and we have had to double our node count.
@johnjeffers Thanks for opening this ticket. I successfully reproduced the issue following your steps. We've confirmed this occurs on EKS 1.34 Bottlerocket AMI specifically due to kernel version 6.12. Earlier Kubernetes versions (pre-1.33) are unaffected as they use kernel 6.1. Our team is currently investigating the kernel-related issue and will provide an update soon. Thank you for your patience.
@gthao313 Thank you! Do you have any idea of when this might be fixed? It's costing us several thousand dollars per month to have to run smaller nodes to avoid this problem.
@johnjeffers. The timeline is still being determined. We'll provide updates when we have more concrete figures. Thanks for your patient.
This issue wasn't present in EKS 1.33 AMIs, and only started after upgrading to EKS 1.34 AMIs.
Could you please share which EKS 1.33 AMIs you were using? Our investigation has shown that this issue only occurs with ARM-based EKS images for versions 1.33 and 1.34.
@jackkleeman that sounds to me like standard EC2 limits, not this bug. With the problem I described it was limited to 15 volumes, whereas 24 is about what I would expect for EC2.
BTW, it seems like this problem is limited to 6g instances. I switched to 7g and I don't see the problem on those. I've also heard (but not yet tested) that the problem is solved with Bottlerocket 1.51.0
@johnjeffers the shared limit would be 27 (28 - instance eni), with root volume and an extra branch ENI I should be allowed 25 pods with volumes (26 volumes overall), but I can only schedule 23 pods (24 volumes). The 25th volume attaches, but the nvme device times out and /dev/xvdax doesn't show up with the same csi error as you. Also detaching hangs and I have to force detach every time.
AWS support directed me to your issue, btw, and they have reproduced what I'm seeing on m6g.xlarge
Your 15 limit - is that a different node type with a lower limit, or are you using a bunch of ENIs for pod networking?
Good to know this is a m6 issue. I'll exclude those nodes from my cluster.
@jackkleeman Oh, interesting! I was using r6g.xlarge, which should also have the same limit of 27. We use pod security groups so the ENI trunk for that takes up one of the slots, but aside from that, nothing unusual.
very interesting that for you the issue kicks in at nvme16 and for me at nvme24! do you also see issues detaching the volumes that went over the limit?
I vaguely remember that, yeah, but I haven't dealt with it since October, when I switched to smaller instance sizes so we didn't hit the limit, so I can't say for sure.
I'm seeing this issue as well after upgrading to EKS 1.34 but it kicks in for me at nvme15 for c6gd.xlarge (I've not seen it in other instances for now). I even raised an issue in aws-ebs-csi-driver repo but maybe it should be addressed here.
EKS version: v1.34.1-eks-113cf36
Kernel Version: 6.12.55
OS Image: Bottlerocket OS 1.51.0 (aws-k8s-1.34)
