How do I use Amazon S3 for storage?

[vtardia]

Hi, what modifications are required to upload asset files to S3? It is a direct upload or pass-through upload? Thanks

[ArekSredzki]

Hi @vtardia. I have not performed those modifications myself, but it should be fairly trivial. Simply replace skipper-disk with skipper-s3 throughout the application. That will give you an implementation which uses pass-through uploads and downloads. You could make further modifications to allow for direct downloads from s3. That cannot be said for uploads because the server must hash certain update types (.nupkg).

[vtardia]

Hi @ArekSredzki,

this is what I have in my sails.config.files, I'm in development env on my local machine:

module.exports.files = {
  maxBytes: 524288000,
  adapter: require('skipper-s3'),
  key: process.env.S3_API_KEY,
  secret: process.env.S3_API_SECRET,
  bucket: process.env.S3_BUCKET,
  region: process.env.S3_REGION
};

The env vars are parsed correctly, the bucket has read/write access for that credentials (tested with Transmit). But when I upload an asset, the server crashes after a minute or so:

events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: Request aborted
    at IncomingMessage.onReqAborted (/path/to/release-server/node_modules/skipper/node_modules/multiparty/index.js:175:17)
    at emitNone (events.js:67:13)
    at IncomingMessage.emit (events.js:166:7)
    at abortIncoming (_http_server.js:280:11)
    at Socket.serverSocketCloseListener (_http_server.js:293:5)
    at emitOne (events.js:82:20)
    at Socket.emit (events.js:169:7)
    at TCP._onclose (net.js:469:12)

What can it be?

[ArekSredzki]

@vtardia I'll provide a full answer later but the issue here is that the config file should not require the actual adapter module, instead you should just specify the name.

Instead you must modify api/services/AssetService.js to use skipper-s3

[vtardia]

@ArekSredzki I've tried modifying only AssetService.js: the assets are uploaded locally and I receive an S3 error (missing key) when I try to delete an asset.

[ArekSredzki]

@vtardia how exactly did you modify it? You must pass in authentication credentials of course

[vtardia]

@ArekSredzki I managed to make it work :-) I've modified AssetService.js like this:

var SkipperDisk = require('skipper-s3');
[...]
var s3Options = {
  key: process.env.S3_API_KEY,
  secret: process.env.S3_API_SECRET,
  bucket: process.env.S3_BUCKET,
  region: process.env.S3_REGION || undefined,
  endpoint: process.env.S3_ENDPOINT || undefined,
  token: process.env.S3_TOKEN || undefined
}
[...]
var fileAdapter = SkipperDisk(s3Options); // every time fileAdapter is created

And config/files.js like this:

module.exports.files = {
  // Maximum allowed file size in bytes
  // Defaults to 500MB
  maxBytes: 524288000,
  // The fs directory name at which files will be kept
  adapter: require('skipper-s3'),
  key: process.env.S3_API_KEY,
  secret: process.env.S3_API_SECRET,
  bucket: process.env.S3_BUCKET,
  region: process.env.S3_REGION || undefined,
  endpoint: process.env.S3_ENDPOINT || undefined,
  token: process.env.S3_TOKEN || undefined
};

On a staging DigitalOcean server, both upload and delete features are working properly, on my local machine I have problems uploading but I think it's due to my ISP.

[shoaibmerchant]

Thanks @vtardia

@ArekSredzki it would be great if this could be part of the core module

[ArekSredzki]

I'll incorporate this slightly differently soon thanks!

[kennyjwilli]

Any update on this?

[ArekSredzki]

no, sorry

[matts2s]

I have configured my instance as @vtardia indicated and while I'm able to upload new content to S3, I get errors on download and delete. I am able to download and delete the same item via aws s3api using the same key/secret. Any ideas on a fix?

[ArekSredzki]

@matts2s in 1.2/1.3 I stopped using skipper for serving files and deleting them because of unnecessary overhead (if one is using skipper-disk)

Take a look at AssetService.js there you'll be able to make the necessary changes

• file serving: instead of streaming the file from the release server, redirect the request to the s3 asset with a 307 response code. Note that 307 would be better than 301 since you want to log the download action on your server, and the asset that is actually served can change with the uri staying the same.
• deletion: just change it back to using skipper

[abacigil]

Hi, I forked and added S3 support, it only supports S3 for storing assets. CloudFront is also available for CDN.

https://github.com/abacigil/electron-release-server

[develar]

@abacigil Maybe you can file PR?

[hdn8]

Another option here is to use s3-fuse and mount a bucket as a disk, although of course better if it's directly integrated in the code for production.

[PierBover]

Hey

Is there any guide on how to use S3 with this project?

@abacigil we tried using your fork but we got a make error unrecognized option -static. I'd post the issue in your repo but it seems those are disabled...

[miguelangelvlc94]

Any update on this?

[ArekSredzki]

@DustinBrett It would be great to merge this in for out of the box configurable support