Lambda

[trailway]

Have you considered using lambda to rapidly copy files from S3 to EBS?

[mwkorver]

Did you mean to say lambda to EC2?

[trailway]

I have been thinking of a system which caches S3 data in high performance storage which can be accessed as a file system from any number of EC2 instances.

Key elements:

• Use Lambda instances to perform any work required, e.g. synchronization.
• Store the data in EFS (still in preview).
  • EFS is designed to support high scalability and does not require you to create EC2 instances to serve the data.
  • As this project has done, this would be a limited size cache using an LRU algorithm.
  • Locking at the file level could be done using Dynamo or some other mechanism.
• In order to determine if data in EFS is stale, i.e. S3 has been updated, one can store the edata tag in EFS and use that to compare to what is in S3. If they are different then the cache is out of date.

Just sharing some thoughts.

It is really a shame that Amazon does not provide this system because I think that this pattern is repeated across many applications.