flatcar
ISCSI not working in kubernetes
Description
When attempting to mount an iscsi volume in a kubernetes pod it hangs forever.
Impact
We aren't able to mount ISCSI inside kubernetes pods.
Environment and steps to reproduce
- Set-up: Rancher Kubernetes v1.31.12+rke2r1, flatcar 4081.2.0 or later, mix of vmware vm and bare metal. iscsid and iscsi services both started
- Task: Applying pod manifest that mounts iscsi volume.
- Action(s): apply pod manifest that mounts iscsi volume
- Error: Pod hangs forever in ContainerCreateing state
Expected behavior
container should get created and mount iscsi volume.
Additional information
I have created the following overview of affected flatcar versions:
| version | state | comment |
|---|---|---|
| flatcar-3815.2.0 | working | |
| flatcar-3975.2.0 | working | |
| flatcar-3975.2.2 | working | |
| flatcar-4081.2.0 | broken | iscsi 2.1.10 introduced |
| flatcar-4081.2.1 | broken | |
| flatcar-4081.3.6 | broken | lts |
| flatcar-4152.2.2 | broken | |
| flatcar-4230.2.1 | broken |
OpenISCSI has a later version than what is shipped in flatcar, https://github.com/open-iscsi/open-iscsi/releases/tag/2.1.11. The changelog mentions the following:
mikechristie (1): iscsid: Fix hang during login with scan=manual (#485)
That issue metions a commit that introduced the regression, https://github.com/open-iscsi/open-iscsi/pull/485, https://github.com/open-iscsi/open-iscsi/commit/48e683c858fa3495aa80402572e1878d057d0399, which was introduced in 2.1.10.
I believe that bumping iscsi to 2.1.11 probably fixes the issues we're having.
I took a look in the beta channel and saw 4344.1.0 updated to 2.1.11. After testing it i can confirm that the issue seems to be resolved.
Are there any timeline on when we can expect this version of Open-ISCSI to reach flatcar stable?
Hi @R0flcopt3r thanks for the detailed description and good job on the investigation. The next release (in October) is planning a new maintenance release for Stable. I'd expect a new major release in November then. I'll let @sayanchowdhury to confirm.
That said, we can investigate to backport ISCSI-2.1.11 to Stable as well. What do you think @krnowak ?
Yeah, bumping sounds good. Filed a PR, waiting for CI to finish: https://github.com/flatcar/scripts/pull/3324
