onestop icon indicating copy to clipboard operation
onestop copied to clipboard

EPIC: Archive workflow support

Open dneufeldcu opened this issue 4 years ago • 2 comments

Summary

As a a data archive
I want to be able to preserve data for the long-term at a reasonable cost
So that I can feel confident in cloud based archival processes

Description

Move files landing in s3 to glacier based on temporal and parentId rules. Be able to restore the data to s3. Understand any AWS limits.

Tasks

  • [x] Design session and documentation of discussion
  • [x] Know when a file goes into glacier
  • [x] Persist the identifier and location of the file
  • [x] Be able to retrieve the file from glacier

Intelligent Storage

  • https://docs.google.com/document/d/1ga10PGamia4Y7-Hl4qWFNJbzr-QlGuPilBNv14y6SDg/edit#heading=h.bh4kpw8i4ph6

dneufeldcu avatar Nov 03 '20 21:11 dneufeldcu

relevant - https://docs.google.com/document/d/1nxYJD4tk6yfT3GdGNHCR8lH8lfYRO4C364ZrJ68NVVk/edit?usp=sharing

caesterlein avatar Nov 05 '20 20:11 caesterlein

Currently we are using the 'glacier' resource type in boto3. This should be switched to s3 resource as glacier api will be deprecated.

dneufeldcu avatar Nov 30 '20 21:11 dneufeldcu