ceph-csi
ceph-csi copied to clipboard
ceph
Research if [csi-driver-iscsi](https://github.com/kubernetes-csi/csi-driver-iscsi) works on Windows.
The effort here is to understand csi driver capabliities against Windows worker nodes.
I am capturing the details here for now, later it can be converted to sub tasks/issues:
https://hackmd.io/oYCS4DviSY2_TTMLft7YPA
Instead of https://github.com/cloudbase/wnbd ?
The experiment here aims an alternate way for mapping /mounting RBD devices in Windows compared to winbd. This will be purely based on CSI protocol spec.
Instead of https://github.com/cloudbase/wnbd ?
NBD exporting and consuming is indeed an alternative. Ideally there should be a csi-driver that supports it on both Linux and Windows. Because csi-driver-iscsi exists, and Ceph can export RBD images over iSCSI already, this might be a relatively simple approach.
As far as my research goes the solution here would be :
- Run Windows Nodes and make it part of the kube cluster
- Run
csi-proxyinside the Windows worker node to facilitate the CSI actions on the windows node - Then deploy
csi-driver-iscsias the CSI driver. That should help us to get the ISCSI mounts working, rest would be the experiment on exporting RBD volumes as iscsi shares and making it accessible in the nodes
Research Doc : https://hackmd.io/oYCS4DviSY2_TTMLft7YPA
From the restrictions listed above, it looks like we can not do "block device mapping" to windows containers. What that means to us ? it looks like we can not expose PVC with block mode to windows containers . May be most of the users are fine with filesystem mode, but listing it out the limitation.
Listing limitations is definitely good!
Please make sure to add links to the documentation and other resources where you found the information in your HackMD notes.
Listing limitations is definitely good!
:+1:
Please make sure to add links to the documentation and other resources where you found the information in your HackMD notes.
Sure @nixpanic , updating the details over there :+1:
https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/
This is good to mention. The table "Support matrix" seems to come from this doc.
As far as my research goes the solution here would be :
1. Run Windows Nodes and make it part of the kube cluster 2. Run `csi-proxy` inside the Windows worker node to facilitate the CSI actions on the windows node 3. Then deploy `csi-driver-iscsi` as the CSI driver. That should help us to get the ISCSI mounts working, rest would be the experiment on exporting RBD volumes as iscsi shares and making it accessible in the nodes
It seems that, even though csi-proxy surface iscsi interface, csi-driver-iscsi lack support for its windows version. If thats done, ideally we should be good to get this combo working.
Research Doc : https://hackmd.io/oYCS4DviSY2_TTMLft7YPA
From the restrictions listed above, it looks like we can not do "block device mapping" to windows containers. What that means to us ? it looks like we can not expose PVC with
blockmode to windows containers . May be most of the users are fine withfilesystemmode, but listing it out the limitation.
As far as my research goes the solution here would be :
1. Run Windows Nodes and make it part of the kube cluster 2. Run `csi-proxy` inside the Windows worker node to facilitate the CSI actions on the windows node 3. Then deploy `csi-driver-iscsi` as the CSI driver. That should help us to get the ISCSI mounts working, rest would be the experiment on exporting RBD volumes as iscsi shares and making it accessible in the nodesIt seems that, even though
csi-proxysurface iscsi interface, csi-driver-iscsi lack support for its windows version. If thats done, ideally we should be good to get this combo working.Research Doc : https://hackmd.io/oYCS4DviSY2_TTMLft7YPA From the restrictions listed above, it looks like we can not do "block device mapping" to windows containers. What that means to us ? it looks like we can not expose PVC with
blockmode to windows containers . May be most of the users are fine withfilesystemmode, but listing it out the limitation.
[Update] Managed to get the windows server running cluster Also with good amount of hacks, I could get the csi-proxy windows version..
Just to mention, the subjected/related projects looks at early stage, so good amount of hacks required to get the experiment going.. :)
Also with good amount of hacks, I could get the csi-proxy windows version..
Do you have these hacks documented? I guess the additional changes should also be reported in the csi-proxy project so that these improvements can benefit others.
It seems there is some support for iSCSI in the csi-proxy project. Did you have a look at that, and got it working too?
Also with good amount of hacks, I could get the csi-proxy windows version..
Do you have these hacks documented? I guess the additional changes should also be reported in the csi-proxy project so that these improvements can benefit others.
Indeed. I have already opened an issue a couple of weeks back in iscsi-csi and made some progress:
https://github.com/kubernetes-csi/csi-driver-iscsi/issues/44 https://github.com/kubernetes-csi/csi-driver-iscsi/pull/45
It seems there is some support for iSCSI in the csi-proxy project. Did you have a look at that, and got it working too?
Yes, I went through the project and related code paths and updated the hackmd with the summary ( please check last week update and todays update ) ie what I think need to be done to get this working. Would like to have a discussion around this and move on further.
As far as my research goes the solution here would be :
1. Run Windows Nodes and make it part of the kube cluster 2. Run `csi-proxy` inside the Windows worker node to facilitate the CSI actions on the windows node 3. Then deploy `csi-driver-iscsi` as the CSI driver. That should help us to get the ISCSI mounts working, rest would be the experiment on exporting RBD volumes as iscsi shares and making it accessible in the nodes
I had this sitting in the editor for a few days and forgot to hit comment... I still want to send it even though the issue has moved on a bit, to try and provide clarity what's really the situation here:
As far as I understand it, numbers 1 and 2 of these should be the standard ones, and they would be expected to be working, maybe with a few tweaks here and there:
- Adding windows nodes is documented in kubernetes (and openshift), and
- the csi-proxy is specifically made to run on windows nodes to enable using csi drivers (node plugins) there. So running it should not be a big issue either.
But the problem is with item number 3: It is not as easy as that. For every kind of operations csi-proxy needs to support (e.g. iscsi, smb, ...), two things need to happen:
- The csi-proxy needs to implement the functionality, and
- the csi-driver needs to be taught to delegate those operations to the csi-proxy.
If you look at smb side, the csi-proxy implements csi and the csi-driver-smb reaches out to csi-proxy for node-operations on windows nodes. So that is all set. For the iscsi-side however, the situation is this:
- csi-proxy has iscsi capability. (not sure it's complete or correct, but it's there)
- iscsi-driver-iscsi does not delegate to csi-proxy on windows nodes (and it's linux implementation is in https://github.com/kubernetes-csi/csi-lib-iscsi and just calls out to iscsiadm essentially), so this will not work out of the box just yet.
So to sum up: it won't work just yet, and in order to make it work, we need to teach csi-driver-iscsi to delegate to csi-proxy on windows nodes. This is the main task, and then we need to eliminate the rough edges.
It seems that, even though
csi-proxysurface iscsi interface, csi-driver-iscsi lack support for its windows version. If thats done, ideally we should be good to get this combo working.
I think that was the short version of what I wrote above, but had to read the code to understand what you were saying. 😉
Research Doc : https://hackmd.io/oYCS4DviSY2_TTMLft7YPA From the restrictions listed above, it looks like we can not do "block device mapping" to windows containers. What that means to us ? it looks like we can not expose PVC with
blockmode to windows containers . May be most of the users are fine withfilesystemmode, but listing it out the limitation.[Update] Managed to get the windows server running cluster Also with good amount of hacks, I could get the csi-proxy windows version..
As @nixpanic mentioned, would be great to document these hacks instead of mentioning that hacks were needed. 😄
Just to mention, the subjected/related projects looks at early stage, so good amount of hacks required to get the experiment going.. :)
............
So to sum up: it won't work just yet, and in order to make it work, we need to teach csi-driver-iscsi to delegate to csi-proxy on windows nodes. This is the main task, and then we need to eliminate the rough edges.
It seems that, even though
csi-proxysurface iscsi interface, csi-driver-iscsi lack support for its windows version. If thats done, ideally we should be good to get this combo working.I think that was the short version of what I wrote above, but had to read the code to understand what you were saying. wink
Exactly @obnoxxx . indeed this need good amount of work! and thats the current state. I updated the hackmd and also discussed in detail in CSI call.
To elaborate some more on this point: The current iscsi csi driver indeed lack the windows support. When we say it lack the support, first of all it has to run inside the windows node, and to support iscsi server operations , the csi iscsi driver has to be implement the csi proxy APIs which I have listed in the issue referenced in earlier comment. Just to mention, it has to implement APIs like below
rpc AddTargetPortal(AddTargetPortalRequest)
returns (AddTargetPortalResponse) {}
// DiscoverTargetPortal initiates discovery on an iSCSI target network address
// and returns discovered IQNs.
rpc DiscoverTargetPortal(DiscoverTargetPortalRequest)
returns (DiscoverTargetPortalResponse) {}
...
// ConnectTarget connects to an iSCSI Target
rpc ConnectTarget(ConnectTargetRequest) returns (ConnectTargetResponse) {}
// DisconnectTarget disconnects from an iSCSI Target
rpc DisconnectTarget(DisconnectTargetRequest)
returns (DisconnectTargetResponse) {}
Making/teaching iscsi csi driver to run and support windows version need effort and support from community. We have to see how this work out against release plans..etc. The linux part also has to be scrutinized with more testing..etc. So there is good amount of work left to get this going.
Research Doc : https://hackmd.io/oYCS4DviSY2_TTMLft7YPA From the restrictions listed above, it looks like we can not do "block device mapping" to windows containers. What that means to us ? it looks like we can not expose PVC with
blockmode to windows containers . May be most of the users are fine withfilesystemmode, but listing it out the limitation.
Thats my understanding too.
[Update] Managed to get the windows server running cluster Also with good amount of hacks, I could get the csi-proxy windows version..
As @nixpanic mentioned, would be great to document these hacks instead of mentioning that hacks were needed. smile
The experiments were carried out in a loaned cluster with very short lived life ( a few hours left before it auto decommissioned) and was facing issues here and there to get it going , also it was first attempt on windows, so hiccups here and there. will try out once again and document it correctly, not sure all those were required though.
Just to mention, the subjected/related projects looks at early stage, so good amount of hacks required to get the experiment going.. :)
Thats the summary so far !
[Status update and some findings after going through various github issues/code/PR..etc on this topic] iow, where we are at:
- Good progress on having first release of
csi-iscsi-driverwhich include the Linux part of Node Controller. Along with other PRs and work, the release is getting close:
Have to address the comments along with few other things https://github.com/kubernetes-csi/csi-driver-iscsi/pull/45#issuecomment-848625777
NAME READY STATUS RESTARTS AGE
pod/csi-iscsi-node-7gwkp 3/3 Running 0 15m
pod/csi-iscsi-node-8dvcj 3/3 Running 0 15m
pod/csi-iscsi-node-b26rh 3/3 Running 0 15m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubelet ClusterIP None <none> 10250/TCP,10255/TCP,4194/TCP 3h43m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/csi-iscsi-node 3 3 3 3 3 kubernetes.io/os=linux 15m
[hchiramm@localhost csi-driver-iscsi]$
[hchiramm@localhost csi-driver-iscsi]$ kubectl get csidriver
NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH MODES AGE
iscsi.csi.k8s.io false false false <unset> false Persistent 16m
[hchiramm@localhost csi-driver-iscsi]$
[hchiramm@localhost csi-driver-iscsi]$ kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pv-name 1Gi RWO Delete Bound test/iscsi-pvc 3m22s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/iscsi-pvc Bound pv-name 1Gi RWO 4s
[hchiramm@localhost csi-driver-iscsi]$
- configuring WMCO : The deployment has to be done using the Red Hat Shipped version of WMCO by following the process mentioned at https://github.com/openshift/windows-machine-config-operator/blob/master/docs/HACKING.md#build . Pretty much got it working but the WMCO pod was getting into
crashloopbackdue to a missing thing in the network configuration in my OCP cluster running on AWS with 4.8.0 build.
Last State: Terminated
Reason: Error
Message: system:serviceaccount:openshift-marketplace:wmco\" cannot get resource \"networks\" in API group \"config.openshift.io\" at the cluster scope","errorVerbose":"networks.config.openshift.io \"cluster\" is forbidden: User \"system:serviceaccount:openshift-marketplace:wmco\" cannot get resource \"networks\" in API group \"config.openshift.io\" at the cluster scope\nerror getting cluster network object\ngithub.com/openshift/windows-machine-config-operator/pkg/cluster.getNetworkType\n\t/build/windows-machine-config-operator/pkg/cluster/config.go:250\ngithub.com/openshift/windows-machine-config-operator/pkg/cluster.networkConfigurationFactory\n\t/build/windows-machine-config-operator/pkg/cluster/config.go:169\ngithub.com/openshift/windows-machine-config-operator/pkg/cluster.NewConfig\n\t/build/windows-machine-config-operator/pkg/cluster/config.go:90\nmain.main\n\t/build/windows-machine-config-operator/main.go:89\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\nerror getting cluster network type\ngithub.com/openshift/windows-machine-config-operator/pkg/cluster.networkConfigurationFactory\n\t/build/windows-machine-config-operator/pkg/cluster/config.go:171\ngithub.com/openshift/windows-machine-config-operator/pkg/cluster.NewConfig\n\t/build/windows-machine-config-operator/pkg/cluster/config.go:90\nmain.main\n\t/build/windows-machine-config-operator/main.go:89\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\nerror getting cluster network\ngithub.com/openshift/windows-machine-config-operator/pkg/cluster.NewConfig\n\t/build/windows-machine-config-operator/pkg/cluster/config.go:92\nmain.main\n\t/build/windows-machine-config-operator/main.go:89\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"main.main\n\t/build/windows-machine-config-operator/main.go:91\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"}
Exit Code: 1
Started: Wed, 26 May 2021 23:32:22 +0530
Finished: Wed, 26 May 2021 23:32:22 +0530
Ready: False
Restart Count: 9
Requests:
cpu: 10m
memory: 50Mi
Liveness: exec [grpc_health_probe -addr=:50051] delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [grpc_health_probe -addr=:50051] delay=5s timeout=5s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
..
Above is with custom images or newly built images:
Normal Pulled 24m kubelet, ip-10-0-135-111.ec2.internal Successfully pulled image "quay.io/humble/wmco:3.0.0" in 401.153338ms
Normal Pulled 24m kubelet, ip-10-0-135-111.ec2.internal Successfully pulled image "quay.io/humble/wmco:3.0.0" in 436.096547ms
Normal Pulled 23m kubelet, ip-10-0-135-111.ec2.internal Successfully pulled image "quay.io/humble/wmco:3.0.0" in 420.653162ms
Once this is corrected, I believe our Windows Machine configuration should be fine.
[Other findings]
-
It has confirmed that, exporting block device to the windows container is not possible at this moment due to an issue in OCI runtime mount specs. So indeed its a limitation for windows containers.
-
csi-proxy layer faciilates the operation on behalf of windows csi containers to avoid any vulnerabilty or security which can arise if the drivers directly started to talk to the windows server.
[Quick update]
I could get the windows working finally!
[hchiramm@localhost windows-machine-config-operator]$ oc get pods
NAME READY STATUS RESTARTS AGE
windows-machine-config-operator-6b4656684-d74fj 1/1 Running 0 2m12s
windows-machine-config-operator-registry-server-5b5b864596xq7zg 1/1 Running 0 2m34s
[hchiramm@localhost windows-machine-config-operator]$
[Status update]
The iscsi experiments are going on heavily to get iscsi csi project in a better shape. Many hiccups due to the fact that, it was never tested in real kube cluster on both codewise or setup wise. Solving one by one and making progress. Will continue experimenting on this as this node part is very important in the workflow of this effort.
Finally able to get the mount part working on ISCSI side: :+1:
I0603 07:11:36.907283 7 mount_linux.go:405] Attempting to determine if disk "/dev/disk/by-path/ip-10.70.53.171:3260-iscsi-iqn.2015-06.com.example.test:target1-lun-1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/by-path/ip-10.70.53.171:3260-iscsi-iqn.2015-06.com.example.test:target1-lun-1])
I0603 07:11:36.925643 7 mount_linux.go:408] Output: "DEVNAME=/dev/disk/by-path/ip-10.70.53.171:3260-iscsi-iqn.2015-06.com.example.test:target1-lun-1\nTYPE=ext4\n", err: <nil>
I0603 07:11:36.925721 7 mount_linux.go:298] Checking for issues with fsck on disk: /dev/disk/by-path/ip-10.70.53.171:3260-iscsi-iqn.2015-06.com.example.test:target1-lun-1
I0603 07:11:37.028261 7 mount_linux.go:394] Attempting to mount disk /dev/disk/by-path/ip-10.70.53.171:3260-iscsi-iqn.2015-06.com.example.test:target1-lun-1 in ext4 format at /var/lib/kubelet/pods/e64ff6a0-fc0a-4f32-a947-04cd873974ce/volumes/kubernetes.io~csi/static-pv-name/mount
I0603 07:11:37.028292 7 mount_linux.go:146] Mounting cmd (mount) with arguments (-t ext4 -o rw,defaults /dev/disk/by-path/ip-10.70.53.171:3260-iscsi-iqn.2015-06.com.example.test:target1-lun-1 /var/lib/kubelet/pods/e64ff6a0-fc0a-4f32-a947-04cd873974ce/volumes/kubernetes.io~csi/static-pv-name/mount)
I0603 07:11:37.086370 7 utils.go:53] GRPC response: {}
I0603 07:11:41.058886 7 utils.go:47] GRPC call: /csi.v1.Identity/Probe
I0603 07:11:41.058968 7 utils.go:48] GRPC request: {}
I0603 07:11:41.059147 7 utils.go:53] GRPC response: {}
The POD runs successfully:
[root@dhcp53-171 csi-driver-iscsi]# kubectl get pods |grep task
task-pv-pod 1/1 Running 0 39s
[root@dhcp53-171 csi-driver-iscsi]#
Unfortunately the unmountis failing which I find as the final blocker for the iscsi csi driver to be in a consumable state, refer https://github.com/kubernetes-csi/csi-driver-iscsi/pull/45#issuecomment-853807843. Debugging it and the RCA is known at this time though.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.
Unfortunately the unmountis failing which I find as the final blocker for the iscsi csi driver to be in a consumable state, refer kubernetes-csi/csi-driver-iscsi#45 (comment). Debugging it and the RCA is known at this time though.
Further experiments were carried on this and various issues are noticed in the driver due to broken library which iscsi driver currently consumes for its operations. Most of the issues are fixed and few are still left to fix. There is a refactoring happening on the library and after that, it has to be retested.
In short, this is getting good pace https://github.com/kubernetes-csi/csi-lib-iscsi/pull/29#issuecomment-918826158 and tagetted for GA at 1.23 timeframe.
[Update]
Finally after many discussions and experiments, https://github.com/kubernetes-csi/csi-lib-iscsi/pull/29 is merged. this should give us decent stable platform to continue on the iscsi driver. Will refresh and update the progress here :+1:
One great update here : Finally kubernetes-csi ISCSI driver get its first release https://github.com/kubernetes-csi/csi-driver-iscsi/releases/tag/v0.1.0 :tada: .. one more step ahead on this effort !
There going to be another iscsi release with 1.24 kube with more fixes..etc. With that in place, we will be able to continue on this effort, meanwhile I am removing this from the 3.6 tracker.
v1.0.0 of iscsi csi-driver is yet to be available https://github.com/kubernetes-csi/csi-driver-iscsi/ . keeping this issue tracker for 3.8
