csi: Cannot create shared volume when setting parameter shared=true

[lpabon]

Is this a BUG REPORT or FEATURE REQUEST?: BUG

What happened: StorageClass in Kubernetes has the value of shared=true. When a Px volume is created, it is not a shared volume.

What you expected to happen: Even though the CO did not use the CSI Access mode of MULTI_NODE_XXX_WRITER, the storage class did set the VolumeSpec value of shared=true, and so a shared volume should have been created.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

[root@dcex-1-node-1 ~]# kubectl describe sc px-hdd-ha2-shared
Name:            px-hdd-ha2-shared
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"px-hdd-ha2-shared","namespace":""},"parameters":{"ephemeral":"false","fs":"xfs","priority_io":"low","repl":"2","shared":"true"},"provisioner":"com.openstorage.pxd"}

Provisioner:           com.openstorage.pxd
Parameters:            ephemeral=false,fs=xfs,priority_io=low,repl=2,shared=true
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

[root@dcex-1-node-1 ~]# kubectl apply -f px-pvc.yaml
persistentvolumeclaim "px-pvc-test2" created

[root@dcex-1-node-1 ~]# kubectl describe pvc px-pvc-test2
Name:          px-pvc-test2
Namespace:     default
StorageClass:  px-ssd-ha2-shared
Status:        Bound
Volume:        pvc-19ddcb0eb71c11e8
Labels:        <none>
Annotations:   control-plane.alpha.kubernetes.io/leader={"holderIdentity":"aa8ac210-b219-11e8-8a60-9e49417cca2f","leaseDurationSeconds":15,"acquireTime":"2018-09-13T06:13:31Z","renewTime":"2018-09-13T06:13:48Z","lea...
               kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volume.beta.kubernetes.io/storage-class":"px-ssd-ha2-shared"},"name":"px-...
               pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
               volume.beta.kubernetes.io/storage-class=px-ssd-ha2-shared
               volume.beta.kubernetes.io/storage-provisioner=com.openstorage.pxd
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      20Gi
Access Modes:  RWX
Events:
  Type    Reason                Age                From                         Message
  ----    ------                ----               ----                         -------
  Normal  ExternalProvisioning  11s (x6 over 20s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "com.openstorage.pxd" or manually created by system administrator
[root@dcex-1-node-1 ~]#
[root@dcex-1-node-1 ~]# pxctl v l
ID                      NAME                    SIZE    HA      SHARED  ENCRYPTED       IO_PRIORITY     STATUS          HA-STATE
997780701893979562      pvc-19ddcb0eb71c11e8    20 GiB  2       no      no              HIGH            up - detached   Detached

Analysis

Bug is due to https://github.com/libopenstorage/openstorage/blob/master/csi/controller.go#L383 .We are doing the correct thing, in that we expect the CO to use the correct CSI annotation to express MULTI_NODE_MULTI_WRITER. The thing is that you have used a Px parameter which the CO does not know about to ask for a Multi-node-multi-writer volume (a shared volume), and so it is overwritten by the CSI spec which does not have that configuration. The code should respect the value in the parameter only when it is true.

[lpabon]

We need to test what happens when the shared volume is to be mounted by the CO, since the CO thinks it is a single_writer.

[lpabon]

This could be harder than we thought. Right now the PV created by the external-provisioner (from Kubernetes-CSI) is always setting the volume to SINGLE_WRITER:

https://github.com/kubernetes-csi/external-provisioner/blob/725706063fd1d5e567a5fff53a6dbe6e38a53d86/pkg/controller/controller.go#L96-L98

[lpabon]

If I fix it in the CSI external-provision, I don't need the fix in OpenStorage.

So... here is how it would work. Instead of setting shared=true, setting the PVC to RWX would automatically create a shared vol. No need to set shared=true in the StorageClass a PVC with RWO would create a non-shared vol. Now the user has control

BUG in external-provisioner (master) https://github.com/kubernetes-csi/external-provisioner/pull/133