Cloud object storage of photos

[damianmoore]

Add support for cloud storage providers (AWS S3, Google Cloud storage) for photo and thumbnail storage. This will allow scaling to an almost infinite number of photos and users. We will support Amazon S3 (-compatible) and Google Cloud Storage (GCS) initially. This ticket does not include anything to do with encryption - there is a separate ticket #59 to add that after. Until encryption is added, all data is publically readable as browser will load the files directly.

AC:

• One bucket configured via environment variables for storing photo data
• One bucket configured via environment variables for storing cache data
• User model extended to have quota_size and current_size (in bytes)
• Library model extended to have quota_size and current_size
• Upload feature to upload to S3 or GCS if Library path starts with s3:// or gcs:// protocols
• Upload feature increases User and Library current_size values on each successful upload (DB transaction)
• Background job to calculate Library and User current sizes once a day to correct any rounding/upload anomilies
• If file is a raw type then raw processor should download from object storage and upload JPEG version to same location
• Thumbnailing job able to retrieve from cloud storage and save to cache location which other analyzers will use
• Ensure other analysis jobs are using 4K thumbnailed versions and trigger thumbnailer first if they don't find cached thumbnail
• Ensure thumbnailer is locking correctly so multiple cloud storage downloads and writes do not occur in parallel
• Thumbnailer URL needs to redirect to cloud location HTTPS endpoint

Dismissed ideas:

• Add Bucket model to assign a bucket to a User.
  • Idea was for quota to be applied for user and they can then create as many libraries as they like. Doesn't seem like Google or AWS allow size quotas to be applied to buckets
  • User should be able
  • Fields: UUID, type (S3 etc), User, path, current_size (needs to be kept updated periodically), max_size (optional)
  • Having separate ID means it can be transferred to another user if required. Also Google recommends not including user IDs in bucket names.
  • If user wishes to create new Library then they can add it to user's existing bucket