Amazon S3 Node (using AWS configuration node)

[hindessm]

trafficstars

Input

• Watch for changes to an S3 bucket:
  • node options
    • bucket - the bucket to watch
    • filepattern - minimatch/glob pattern to filter files on (optional, default include all files)
  • output msg properties
    • payload - the full path of the changed file
    • file - the basename of the changed file
    • event - the type of event either add or delete

Query

• Retrieve a file from an S3 bucket:
• Get S3 file contents
  • node options
    • bucket - bucket to retrieve file from
    • filename - full path of file to retrieve
    • localfilename - local file to write content to (optional - defaults to storing content in payload)
  • msg options
    • bucket - bucket to retrieve file from
    • filename - full path of file to retrieve
    • localfilename - local file to write content to
  • output msg properties
    • payload - the content of the file (deleted if there is an error)
    • bucket - the bucket the file was retrieved from
    • filename - the full path of the file retrieved
    • error - an error object (deleted if the request was successful)

Outputs

• Write to a file (create/replace)
  • node options
    • bucket - a bucket to store files in
    • filename - filename of the remote s3 file
    • localFilename - local filename containing the content to be uploaded to s3
  • msg options
    • bucket - override the node configured bucket
    • filename - filename of the remote s3 file
    • localFilename - local filename containing the content to be uploaded to s3
    • payload - the content to be uploaded (localFilename takes priority)

Authentication

Users must register their own app at http://aws.amazon.com/ and paste the access key id and secret access key into the node.

[knolleary]

Given the breadth of Amazon's services and apis, I would expect multiple Amazon Input/Output nodes logically grouped around each service, with the proper shared configuration (ie credentials).

It might be better to create separate sub-epics for the difference services, otherwise this one would get unmanageable in size.

[hindessm]

I'm initially going to write an AWS node to maintain the credentials (and potentially other shared but overrideable configuration like region) and an S3 node that uses this. Perhaps this can be one epic since it is necessary to have a concrete use of the configuration node to really prove it works. I'll rename this issue accordingly.

Then we can have other epics for product api and other services.

Does that seem reasonable?

[knolleary]

:thumbsup:

[dceejay]

Yes - we have a contributed Amazon (storage node) in node-red-nodes/storage/ddb
(no idea why it's not in it's own dir... indeed looking at it I have no idea what it does...)

[knolleary]

I suggest we move the DDB node over to this repo along with the other aws nodes we'll end up with. We can check the download stats of the DDB node's npm to see if it's being used at all.

[hindessm]

I didn't notice the ddb one so I've already written an aws node which uses the new credentials interface. The only differences are mine is just called "aws" rather than "aws credentials" since I intended for it to have a default region in it and I called the credentials by the name "accessKeyId" rather than "accessKey" since that is what Amazon documentation uses.

I also have an AWS property on the aws node to get the configured AWS top-level object rather than having to do that in each module.

I could add the AWS property to the existing "aws credentials" node and rename the accessKey => accessKeyId but I'm not sure that is the right thing to do.

[vielmetti]

One more thing to add to this node would be "endpoint", per

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Endpoint.html

Allowing the user to override endpoint would enable the use of S3-compatible endpoints like Minio, a la

https://docs.minio.io/docs/how-to-use-aws-sdk-for-javascript-with-minio-server

[dceejay]

I don't use this node day to day so a PR would be most welcome.