aws-ebs-csi-driver
aws-ebs-csi-driver copied to clipboard
kubernetes-sigs
CSI support attach a volume to multiple instances with amazon EBS Multi-Attach
Describe the solution you'd like in detail https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volumes-multi.html
Hey, there is no progress on this feature yet as we still need the design for this
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
Hey, there is no progress on this feature yet as we still need the design for this
Are there any design discussions happening somewhere else? @leakingtapan
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale
Is there any progress? There is an open issue on stackoverflow https://stackoverflow.com/a/67210073/7755174
Multi-attached EBSs have several limitations. I'm wondering how people are expecting to actually use this? It won't be as simple as attaching a PVC to multiple pods:
Standard file systems, such as XFS and EXT4, are not designed to be accessed simultaneously by multiple servers, such as EC2 instances. Using Multi-Attach with a standard file system can result in data corruption or loss, so this is not safe for production workloads. You can use a clustered file system to ensure data resiliency and reliability for production workloads.
Multi-Attach enabled volumes do not support I/O fencing. I/O fencing protocols control write access in a shared storage environment to maintain data consistency. Your applications must provide write ordering for the attached instances to maintain data consistency.
From https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volumes-multi.html.
Seems like it would only be useful for specialized purposes - if people tried using multi-attached EBSs as shared PVCs with typical filesystems they are going to run into a world of pain.
Those caveats are common to multiple writers on any shared block storage aren't they?
Our application is to write once, detach the writer and then attach read only to many. So a clustered FS is not needed.
+1 to @jsravn above.
What use cases is everyone trying to solve? How you plan to use it?
We use Tekton as part of our CI/CD system. Tekton allows you to create workspaces using PVCs. When a pipeline executes, Tekton creates an affinity assistant if the pipeline requires a workspace. This affinity assistant is used to run all pods that the tasks in the pipeline create on the same instance that the EBS volume is bound to, for obvious reasons. During scheduling, the affinity assistant resource requests are extremely small and it is unaware of the resource requests of the pods that the pipeline will create. So when our build nodes are close to capacity, and the affinity assistant is scheduled to one of these nodes, the subsequent tasks take a long time to provision as it needs to wait for resources to free up.
If we had multi-attach EBS, we would have no need for the affinity assistant (our pod templates would restrict it to running in the same zone, of course) and this would improve our build times and even out our EC2 utilisation.
Those caveats are common to multiple writers on any shared block storage aren't they?
Our application is to write once, detach the writer and then attach read only to many. So a clustered FS is not needed.
Even ext4 writes back to the journal on readonly mounts. So you're playing with fire and risking your data in a major way. Not a good idea.
If we had multi-attach EBS, we would have no need for the affinity assistant (our pod templates would restrict it to running in the same zone, of course) and this would improve our build times and even out our EC2 utilisation.
No, multi-attach doesn't give you this. It's shared block storage, not a clustered file system that supports multiple writers. Maybe you want K8s to manage/setup something like Redhat GFS for you?
Even ext4 writes back to the journal on readonly mounts. So you're playing with fire and risking your data in a major way. Not a good idea.
"playing with fire and risking your data in a major way" is FUD. It's completely fine to attach multiple readers (with no writers) in a way that is entirely safe:
- Ensure the filesystem was unmounted cleanly
- Make the underlying block device read-only (not strictly needed, but helps avoid mistakes)
- Mount with
ro,noload(noloadis safe because there is no journal to replay — the filesystem was unmounted cleanly)
If you can't ensure these constraints then don't do it. But it's perfectly safe if done properly.
I personally wouldn't risk disabling journalling, even read-only, in the general case (due to nodes crashing/EBS's hanging/etc.)
There are no writes to the journal on a read-only mount even if nodes crash, the block storage hangs, or anything else. The only writes that will ever occur on a read-only mount are on startup to replay the journal, if and only if the filesystem was not unmounted cleanly by the most recent read/write mount.
With extfs there are enormous risks of having multiple readers if any writer exists, but (assuming the most recent read/write mount was unmounted cleanly) there are zero risks of multiple readers if no writer exists. It's not that niche a case, honestly.
I would think that this driver should focus on the block storage part? As far as I understand it, fsType can be anything supported by the node, so for those that do need read/write, quite a few Linux clustered filesystems should immediately be possible if this driver can supply the multi-attached block storage.
Sorry, deleted my comment because I felt I was going way off topic.
I agree the driver should just focus on the block storage part.
AFAIK there is no "fsType" that would support it right now - they usually all require supporting infrastructure like Redhat DLM for GFS. There is also the question of I/O fencing which multi-attach doesn't support. It's likely more work will have to be done on the K8s storage side to provide the functionality people are expecting.
I'd think right now this would just be used in block storage mode and users have to manage it themselves by setting up the infra needed. Or carefully manage the mount lifecycle like you suggest.
Any updates on this issue? Are there plans to add this feature of supporting multiAttach in the CSI driver? I see there is a PR raised but in a closed state now. https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/1177 If needed, I can take up and raise a PR.
Is there a particular reason why, for pods requiring a standard filesystem, that EFS / NFS is not a viable solution to the problem. It seems to be way less complicated, have far fewer pre-conditions (e.g. pre-building volumes, ensuring clean unmouts, etc.) and is fully supported by EKS, EKS Fargate and standalone Kube. Multi-attach is provided largely to meet very specific use cases where software needs concurrent access to manage a block device / filesystem, whereas EFS is really intended to address the broader generic shared filesystem functionality.
Is there a particular reason why, for pods requiring a standard filesystem, that EFS / NFS is not a viable solution to the problem. It seems to be way less complicated, have far fewer pre-conditions (e.g. pre-building volumes, ensuring clean unmouts, etc.) and is fully supported by EKS, EKS Fargate and standalone Kube. Multi-attach is provided largely to meet very specific use cases where software needs concurrent access to manage a block device / filesystem, whereas EFS is really intended to address the broader generic shared filesystem functionality.
One common reason I had in using EBS multi-attach instead of EFS is cost (On a cost perspective only, EFS is ~2-3x more expensive than EBS - io2). In addition, I will also like to perform a lift and shift migration to EKS from VM, replicating what the workload have been using in VMs (multiple EC2 to 1 EBS volume). These workloads are mainly read-only
Another use case for this feature is enabling persistent volumes for Windows worloads, FSx for Windows File Server is not always a choice since you need an Active Directory Domain to even deploy a FSx file system.
