gcp-compute-persistent-disk-csi-driver icon indicating copy to clipboard operation
gcp-compute-persistent-disk-csi-driver copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes-sigs

PD --multi-writer doesn't seem to work with GKE/K8S

Open scanfield opened this issue 3 years ago • 8 comments

TLDR: attaching a --multi-writer PD to 2 GCE VMs works as expected, but if using GKE & exposing the PD as a block device then each pod will not see the other's writes unless the disk is detached and re-attached (e.g. when the pod restarts).

Here's a git repo with reproduction steps: https://github.com/scanfield/mw-repro

I know this is probably on the bleeding edge since --multi-writer is in beta, but it seems like the K8S functionality must work with other types of multi writer volumes.

Here's a slack thread for some context: https://kubernetes.slack.com/archives/C09QZFCE5/p1632980542168500

scanfield avatar Oct 02 '21 04:10 scanfield

/assign @mattcary

Thanks Steven! I'll work on a repro.

mattcary avatar Oct 06 '21 16:10 mattcary

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jan 04 '22 17:01 k8s-triage-robot

/lifecycle frozen

This is still in my queue

mattcary avatar Jan 04 '22 18:01 mattcary

Extremely gentle ping -- this would still be quite useful

scanfield avatar Feb 24 '23 02:02 scanfield

Thanks for the ping, still in my queue :-P

mattcary avatar Feb 24 '23 19:02 mattcary

I looked into this for a bit, but didn't make progress.

I verified your repro (thanks for such a clear example!) and then simultaneously attached the same shared disk to bare vms, and verified that while the devices on the VM were seeing data immediately appear back and forth, those same writes were at the same time not visible from within the k8s pod.

I thought it might have to do with the bind mount k8s is doing. See here for how the kubelet sets up block devices.

But making bind mounts on the bare vm and writing through them didn't have any affect---the writes were immediately visible across different VMs (except the k8s ones).

Hrm.

mattcary avatar Apr 08 '23 00:04 mattcary

Next layer to check could be the container runtime

msau42 avatar Apr 24 '23 22:04 msau42

Yeah good point. Maybe something with mount propagation.

mattcary avatar Apr 24 '23 22:04 mattcary