containers-roadmap [ECS] Support for FSx for Lustre in ECS on EC2 and Fargate

[ECS] Support for FSx for Lustre in ECS on EC2 and Fargate

Open coultn opened this issue 4 years ago • 25 comments

Similarly to our upcoming support for EFS, we are considering support for FSx for Lustre filesystems within an ECS task definition. A sample task definition may look like the following (subject to change as we work on the design):

{
    "family": "my-task-with-fsx",
    "volumes": [
        {
          "name": "myFSxVolume",
          "FSxLustreVolumeConfiguration": {
                "filesystem": "fs-1234"
            }
        }
    ],
    "containerDefinitions": [
        {
           "name": "container-using-fsx",
           "mountPoints": [
               {
                   "sourceVolume": "myFSxVolume",
                   "containerPath": "/mount/fsx",
                   "readOnly": true
               }
        ]
    }
   ]
}

Dec 13 '19 17:12 coultn

Hey @coultn this sounds awesome.

Could you briefly explain why would someone use EFS instead of FSx and vice versa for task definitions in ECS?

Thanks and keep up the good work!

Dec 25 '19 14:12 dimisjim

EFS and FSx for Lustre have different capabilities, performance characteristics, and different pricing models. FSx for Lustre is "is optimized for data processing, with sub-millisecond latencies and throughput that scales to hundreds of gigabytes per second." EFS is oriented towards customers who need a more traditional shared filesystem: "Amazon EFS provides secure access for thousands of connections for Amazon EC2 instances and on-premises servers simultaneously using a traditional file permissions model, file locking capabilities, and hierarchical directory structure via the NFSv4 protocol. "

Jan 03 '20 00:01 coultn

With the EKS CSI driver support for lustre, seems logical that ECS should have support as well.

Jan 07 '20 16:01 unicomp22

Any ETA as to when this (FSx for Luster) will make into the ECS/Fargate supported CSI? We have a lot of containerized use cases that requires high performance data processing.

Feb 12 '20 21:02 linuxus

Does anything prevent me from installing/using fsx for lustre within a firecracker vm guest instance? The same way I would install/use it in an ec2 instance?

Feb 13 '20 19:02 unicomp22

One use case we have for FSx on Fargate is workflows using Step Functions that invoke fargate to dynamically mount an FSx drive to do a task. We do this currently with Batch but I would like to switch to Fargate. We installed Luster drivers on an AMI that we use in Batch and have our code in the docker do the mount.

May 18 '20 16:05 leedahl

Does Fargate support FSx Luster?

Oct 29 '20 23:10 Sidra

Repeating @Sidra's questions from above: Does Fargate support FSx Luster?

Mar 02 '21 16:03 michel-dance

No, it does not.

Mar 02 '21 16:03 coultn

Hi, any updates on this? we would really really love to have this capability. FSx makes a whole lot more sense for our use case than EFS due to price/performance characteristics