Supporting calling via SQS events with backup config in SQS message

[bigpresh]

Hi,

I'm trying to set up a use of pgdump-aws-lambda where it would be triggered by an SQS queue, with the params for the backup run in the queue message.

However, the event that pgdump-aws-lambda sees has the details it would need under Records - e.g. event.Records[0].body containing JSON.

I was thinking of raising a PR so that, if the event being processed has Records, it will, if there is exactly one record, attempt to deserialize that record's Body from JSON and put the values into the event... so that you can queue an SQS message with the PGDATABASE, PGUSER etc to be used, and pgdump would be triggered by SQS and understand how to get that information from the event.

I would assume that you'd configure the triggering to have a batch size of 1, rather than attempt to add a whole lot more complexity in one run of the lambda trying to back up multiple DBs in one execution - I think that would both make things more complex, and in general be a bad idea as you'd be more likely to run in to execution time issues - better to just trigger it off multiple times concurrently.

Does this sound like a reasonable idea?

[bigpresh]

Just as an update, the route I actually went for was to "wrap" the lambda - so the entry point was my code, index.ts in this case, which exports a handler which takes the SQS event, unpacks the details from it (e.g. event.Records[0].body), then creates a fake event object with all the stuff that pgdump-aws-lambda needs and passing that fake event object to it - something like:

// Wrap the pgdump-aws-lambda lamba code

import * as AWS from "aws-sdk";
import { SQSEvent } from "aws-lambda";
import * as pgdumpawslambda from "./pgdump-aws-lambda";

// Get and parse the SQS event that triggered us, pulling out the DB details from `Records` in it,
// create a faked record suitable for pgdump-aws-lambda containing the details from the event along
// with the credentials we fetched, then hand off to pgdump-aws-lambda's handler()
export const handler = async (event: SQSEvent): Promise<any> => {
  const eventBody = JSON.parse(event.Records[0].body);

  // fetch database credentials via SSM and other info

  const fakeEvent = {
    PGHOST: process.env.PGHOST,
    PGDATABASE: eventBody.PGDATABASE,
    PGUSER: dbCreds.username,
    PGPASSWORD: dbCreds.password,
    S3_BUCKET: process.env.S3_BUCKET,
    PGDUMP_PATH: "bin/postgres-13.3",
  };

  console.log(`Call pgdump-aws-lambda passing our faked event:`, fakeEvent);

  // Hand off to pgdump-aws-lamba here
  await pgdumpawslambda.handler(fakeEvent);
};

... with the actual pgdump-aws-lamba code living in its own dir a level below that.