batch processing from public bucket / subscribing to SNS topic

[adokter]

I've been using SCAR to batch process data from a public bucket that posts SNS notifications whenever new files are added

Currently SCAR handles S3 events, but not SNS events. I managed to add SNS functionality in an older version of SCAR (e.g. see here and here), but SCAR has evolved much and I'm no longer sure how to apply a patch myself. Therefore this feature request to be able to subscribe to an SNS topic and be able to batch process public buckets.

Thanks for developing SCAR!

[srisco]

Hi @adokter, sorry for the late reply. We're studying the integration with SNS, but at the moment we are too busy and we think that it may be so specific functionality for your use case (the format of the notifications may vary). At the moment I can give you two ideas to implement it:

• You can fork the FaaS Supervisor and add the functionality here. Currently you can't specify a custom repo for the supervisor in your function config, but if you open a PR we can create you a custom pre-release for testing purposes.
• You can edit your user script in order to process the event as an UnknownEvent by reading it from $INPUT_FILE_PATH (with jq for example), and download the data from the public bucket with aws-cli.

Hope you can find it useful. If you have any doubt don't hesitate to contact us again, this time I will try to answer you faster.

Greetings.

[adokter]

Thanks very much @srisco for the pointers, I will look into it