onestop
onestop copied to clipboard
EPIC: Archive workflow support
Summary
As a | a data archive |
I want to be able to | preserve data for the long-term at a reasonable cost |
So that I can | feel confident in cloud based archival processes |
Description
Move files landing in s3 to glacier based on temporal and parentId rules. Be able to restore the data to s3. Understand any AWS limits.
Tasks
- [x] Design session and documentation of discussion
- [x] Know when a file goes into glacier
- [x] Persist the identifier and location of the file
- [x] Be able to retrieve the file from glacier
Intelligent Storage
- https://docs.google.com/document/d/1ga10PGamia4Y7-Hl4qWFNJbzr-QlGuPilBNv14y6SDg/edit#heading=h.bh4kpw8i4ph6
relevant - https://docs.google.com/document/d/1nxYJD4tk6yfT3GdGNHCR8lH8lfYRO4C364ZrJ68NVVk/edit?usp=sharing
Currently we are using the 'glacier' resource type in boto3. This should be switched to s3 resource as glacier api will be deprecated.