digital-paper-edit-client icon indicating copy to clipboard operation
digital-paper-edit-client copied to clipboard

S3 file upload cloud formation setup

Open emettely opened this issue 5 years ago • 6 comments

Context

We want to upload directly to S3 from the client using the signed URL.

Acceptance Criteria

  • [x] Set up S3 in Cloudformation and deploy in Cosmos (infrastructure repository)

-Make sure upload of files are only done through signed URLs, retrieved from the API- (moved to a different ticket)

Some useful references

emettely avatar Jun 25 '19 14:06 emettely

I'm setting up the S3 as part of the API.

emettely avatar Jun 26 '19 11:06 emettely

The S3 is setup for int and test bundled with the API. The permissions from the API is:

                                "s3:ListBucket",
                                "s3:*Object"

https://github.com/bbc/digital-paper-edit-infrastructure/blob/master/dpe-api/infrastructure/src/main.py

^ This is the bit that has that logic. @jamesdools and @pietrop , we might want to consider what the life-cycle (expire in 1 year or half a year...) of items in the bucket should be. I have it currently commented out - but once we figure out the subdirectory names (e.g. audio, original, preview etc.) we can uncomment + implement the lifecycle policy.

emettely avatar Jun 26 '19 11:06 emettely

@emettely can you add some more details on how this works?

I get there's something added to the cloud formation to give the EC2 instance for the API serve the capability to access the bucket(?)

Do we know more about how do you then access (add or get files from the S3 bucket) from within the express server inside the API? (do we use the AWS Node SDK?)

Also I see the link goes straight into master, was there a PR for this?

And is this still in progress ready for review or done?

pietrop avatar Jun 28 '19 13:06 pietrop

narrowing down this card to be only covering first part of acceptance criteria to do with

  • [x] Set up S3 in Cloudformation and deploy in Cosmos (infrastructure repository)

moving the rest to a separate card https://github.com/bbc/digital-paper-edit-api/issues/4

pietrop avatar Jun 28 '19 14:06 pietrop

To review this (draft)

  1. Use python script to generates cloud formation file
  2. Use Jenkins to upload to AWS (stack job)
  3. see if it gets picked up in int, test etc.. in cosmos
  4. in AWS console you should see the resources with right policies

pietrop avatar Jun 28 '19 15:06 pietrop

https://github.com/bbc/digital-paper-edit-infrastructure/compare/b04120ea8a865557946bfebfe851578e77833212...c33727bfbe9a3afc197356db61c87c55e97c5679

emettely avatar Jun 28 '19 15:06 emettely