Bottlerocket under-reports Ephemeral Storage Capacity

[jonathan-innis]

Image I'm using:

AMI Name: bottlerocket-aws-k8s-1.22-aarch64-v1.11.1-104f8e0f

What I expected to happen:

I expected that the capacity on my worker node would be approximately close to what the actual EBS volume size is to the xvdb mount for my node filesystem.

bash-5.1# df -h
Filesystem       Size  Used Avail Use% Mounted on
/dev/root        904M  557M  285M  67% /
devtmpfs          16G     0   16G   0% /dev
tmpfs             16G     0   16G   0% /dev/shm
tmpfs            6.1G  1.2M  6.1G   1% /run
tmpfs            4.0M     0  4.0M   0% /sys/fs/cgroup
tmpfs             16G  476K   16G   1% /etc
tmpfs             16G  4.0K   16G   1% /etc/cni
tmpfs             16G     0   16G   0% /tmp
tmpfs             16G  4.0K   16G   1% /etc/containerd
tmpfs             16G   12K   16G   1% /etc/host-containers
tmpfs             16G  4.0K   16G   1% /etc/kubernetes/pki
tmpfs             16G     0   16G   0% /root/.aws
/dev/nvme1n1p1   4.3T  1.6G  4.1T   1% /local
/dev/nvme0n1p12   36M  944K   32M   3% /var/lib/bottlerocket
overlay          4.3T  1.6G  4.1T   1% /aarch64-bottlerocket-linux-gnu/sys-root/usr/lib/modules
overlay          4.3T  1.6G  4.1T   1% /opt/cni/bin
/dev/loop1       384K  384K     0 100% /aarch64-bottlerocket-linux-gnu/sys-root/usr/share/licenses
/dev/loop0        12M   12M     0 100% /var/lib/kernel-devel/.overlay/lower
overlay          4.3T  1.6G  4.1T   1% /aarch64-bottlerocket-linux-gnu/sys-root/usr/src/kernels

bash-5.1# lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0          7:0    0 11.6M  1 loop /var/lib/kernel-devel/.overlay/lower
loop1          7:1    0  292K  1 loop /aarch64-bottlerocket-linux-gnu/sys-root/usr/share/licenses
nvme0n1      259:0    0    2G  0 disk 
|-nvme0n1p1  259:2    0    4M  0 part 
|-nvme0n1p2  259:3    0    5M  0 part 
|-nvme0n1p3  259:4    0   40M  0 part /boot
|-nvme0n1p4  259:5    0  920M  0 part 
|-nvme0n1p5  259:6    0   10M  0 part 
|-nvme0n1p6  259:7    0   25M  0 part 
|-nvme0n1p7  259:8    0    5M  0 part 
|-nvme0n1p8  259:9    0   40M  0 part 
|-nvme0n1p9  259:10   0  920M  0 part 
|-nvme0n1p10 259:11   0   10M  0 part 
|-nvme0n1p11 259:12   0   25M  0 part 
`-nvme0n1p12 259:13   0   42M  0 part /var/lib/bottlerocket
nvme1n1      259:1    0  4.3T  0 disk 
`-nvme1n1p1  259:14   0  4.3T  0 part /var
                                      /opt
                                      /mnt
                                      /local

What actually happened:

status:
  addresses:
  - address: 192.168.124.121
    type: InternalIP
  - address: ip-192-168-124-121.us-west-2.compute.internal
    type: Hostname
  - address: ip-192-168-124-121.us-west-2.compute.internal
    type: InternalDNS
  allocatable:
    attachable-volumes-aws-ebs: "39"
    cpu: 15890m
    ephemeral-storage: "1342050565150"
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    hugepages-32Mi: "0"
    hugepages-64Ki: "0"
    memory: 28738288Ki
    pods: "234"
    vpc.amazonaws.com/pod-eni: "54"
  capacity:
    attachable-volumes-aws-ebs: "39"
    cpu: "16"
    ephemeral-storage: 1457383148Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    hugepages-32Mi: "0"
    hugepages-64Ki: "0"
    memory: 31737584Ki
    pods: "234"
    vpc.amazonaws.com/pod-eni: "54"

CAdvisor or something in the BR image appears to be under-reporting the amount of capacity that I have on this worker node.

The ephemeral-storage capacity here is approximately 1457383148Ki ~= 1.35 Ti which is not close to the ~4.3T that the lsblk is reporting.

How to reproduce the problem:

1. Launch a image with the BR AMI that has an EBS volume above > 4TB attached to the /dev/xvdb mount
2. View the Node's capacity when it connects and joins to the cluster using kubectl get node or kubectl describe node