same directory is bind-mounted 32767 times

[iameli]

Moved from https://github.com/k3s-io/k3s/issues/6660.

I have no idea who to report this bug to, so I'm going to duplicate the report a few places. kubernetes: https://github.com/kubernetes/kubernetes/issues/114583 core-dump-handler: https://github.com/IBM/core-dump-handler/issues/119

Environmental Info: K3s Version:

root@dp2426:~# k3s -v
k3s version v1.23.4+k3s1 (43b1cb48)
go version go1.17.5

I have also seen this behavior on a different node running a more recent version

root@dp7744:~# k3s -v
k3s version v1.25.3+k3s1 (f2585c16)
go version go1.19.2

Node(s) CPU architecture, OS, and Version:

root@dp2426:~# uname -a
Linux dp2426 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration: 6 Server Nodes

Describe the bug: I'm running core-dump-handler on a few nodes. When core-dump-handler comes under load — we had a service elsewhere that was malfunctioning and segfaulting many times per second — its directory gets bind-mounted over and over and over and over. I do not know by whom.

mount | grep core
/dev/md1 on /home/data/core-dump-handler/cores type ext4 (rw,relatime,stripe=256)
/dev/md1 on /home/data/core-dump-handler/cores type ext4 (rw,relatime,stripe=256)
/dev/md1 on /home/data/core-dump-handler/cores type ext4 (rw,relatime,stripe=256)
/dev/md1 on /home/data/core-dump-handler/cores type ext4 (rw,relatime,stripe=256)
/dev/md1 on /home/data/core-dump-handler/cores type ext4 (rw,relatime,stripe=256)
/dev/md1 on /home/data/core-dump-handler/cores type ext4 (rw,relatime,stripe=256)
/dev/md1 on /home/data/core-dump-handler/cores type ext4 (rw,relatime,stripe=256)
/dev/md1 on /home/data/core-dump-handler/cores type ext4 (rw,relatime,stripe=256)
/dev/md1 on /home/data/core-dump-handler/cores type ext4 (rw,relatime,stripe=256)
/dev/md1 on /home/data/core-dump-handler/cores type ext4 (rw,relatime,stripe=256)
/dev/md1 on /home/data/core-dump-handler/cores type ext4 (rw,relatime,stripe=256)
[...]

# mount | grep core | wc -l
32767

Steps To Reproduce: No idea how to reproduce this in an isolated environment, but I'll give it a shot as I continue debugging.

Here's core-dump-handler's DaemonSet configuration file and the PVCs that back it. The pertinent volumes section:

      volumes:
      - name: host-volume
        persistentVolumeClaim:
          claimName: host-storage-pvc
      - name: core-volume
        persistentVolumeClaim:
          claimName: core-storage-pvc
[...]
        volumeMounts:
        - mountPath: /home/data/core-dump-handler
          mountPropagation: Bidirectional
          name: host-volume
        - mountPath: /home/data/core-dump-handler/cores
          mountPropagation: Bidirectional
          name: core-volume

Possibly a problem with bind-mounting one directory inside another...?

I'll certainly be opening a report against core-dump-handler but it seems like it must be k8s's bad behavior someplace to create multiple system-level mounts...?