make deploy works but the lambda function doesn't

[kqdtran]

Hey @CodyReichert and the Assertible team, thanks again for this very useful module. I'm having some problem getting it to work. I've changed the settings in the Makefile:

LAMBDA_TEST?=./node_modules/node-lambda/bin/node-lambda
LAMBDA_FUNCTION_NAME=cloudwatch-to-slack
AWS_REGION=us-east-1
AWS_ROLE=arn:aws:iam::XXXXX
AWS_PROFILE=myprofile
AWS_ACCESS_KEY_ID=XXXXX
AWS_ACCESS_KEY_SECRET=XXXXX

and also in the config.js:

kmsEncryptedHookUrl: "<kmsEncryptedHookUrl>", // encrypted slack webhook url
unencryptedHookUrl: "https://hooks.slack.com/services/XXXXX", // unencrypted slack webhook url
slackChannel: "#aws"

(the rest of the settings in config.js are left unchanged.) I then run npm install and make test, and it works perfectly. The four sample notifications were posted to our slack channel. Yay!

Since the current version of the README ends here, I played around and figured that I needed to run make deploy to use this in production (I also added the AWS_ACCESS_KEY_ID and AWS_ACCESS_KEY_SECRET while doing this). So I ran this, and it worked great as well.

=> Moving files to temporary directory
=> Running npm install --production
=> Zipping deployment package
=> Reading zip file to memory
=> Uploading zip file to AWS Lambda us-east-1 with parameters:
{ FunctionName: 'cloudwatch-to-slack-development',
  Code: { ZipFile: <Buffer 50 4b 03 04 14 00 00 00 08 00 77 60 1e 49 0e 26 12 34 f4 01 00 00 28 05 00 00 09 00 1c 00 63 6f 6e 66 69 67 2e 6a 73 55 54 09 00 03 62 ae c5 57 62 ae ... > },
  Handler: 'index.handler',
  Role: 'XXXXX',
  Runtime: 'nodejs4.3',
  Description: 'Better Slack notifications for AWS CloudWatch',
  MemorySize: '128',
  Timeout: '3',
  Publish: false,
  VpcConfig: {} }
=> Zip file(s) done uploading. Results follow

All look good. I wait for a bit but receives no notification on the Slack channel, so I decide to check the AWS Lambda page on the console. I then choose Test => Configure test event => Cloudwatch logs in the sample event template. The test result looks like:

START RequestId: 9073e2c4-6ed2-11e6-874d-837e65e4596f Version: $LATEST
2016-08-30T16:55:36.213Z    9073e2c4-6ed2-11e6-874d-837e65e4596f    sns received:{
  "awslogs": {
    "data": "H4sIAAAAAAAAAHWPwQqCQBCGX0Xm7EFtK+smZBEUgXoLCdMhFtKV3akI8d0bLYmibvPPN3wz00CJxmQnTO41whwWQRIctmEcB6sQbFC3CjW3XW8kxpOpP+OC22d1Wml1qZkQGtoMsScxaczKN3plG8zlaHIta5KqWsozoTYw3/djzwhpLwivWFGHGpAFe7DL68JlBUk+l7KSN7tCOEJ4M3/qOI49vMHj+zCKdlFqLaU2ZHV2a4Ct/an0/ivdX8oYc1UVX860fQDQiMdxRQEAAA=="
  }
}
2016-08-30T16:55:36.252Z    9073e2c4-6ed2-11e6-874d-837e65e4596f    TypeError: Cannot read property '0' of undefined
    at processEvent (/var/task/index.js:223:43)
    at exports.handler (/var/task/index.js:267:5)
END RequestId: 9073e2c4-6ed2-11e6-874d-837e65e4596f
REPORT RequestId: 9073e2c4-6ed2-11e6-874d-837e65e4596f  Duration: 3584.87 ms    Billed Duration: 3600 ms    Memory Size: 128 MB Max Memory Used: 26 MB  
Process exited before completing request

, which indicates that there's some bug with the module (I don't know nodejs, just purely guessing), or that I'm doing something wrong and the notification is not being posted correctly.

Any suggestion would be greatly appreciated, thanks!

[knowroozi]

@kqdtran i think only cloudwatch alarm notifications are supported, not events from logs

[creichert]

@kqdtran Are you saying there was a failed Cloudwatch event and you saw that error in the logs? Where was the event from?

Sorry on the README, I'll get that updated today

[kqdtran]

No prob on the README, thanks for working on it!

Yes, there are lots of Cloudwatch events and we are currently not receiving any notification on Slack (we are via email, however.) What's the best course of action to test that things work after running make deploy?

If it makes any difference, our instances and RDS db are all in a VPC. Do we need to specify the VPC config for this lambda function?

[creichert]

It's likely failing on a call like this:

var snsSubject = event.Records[0].Sns.Subject;

It would help to get the event json (or the shape) if at all possible. I'll run through some tests on this in our infra and see if I can come up with anything.

[kqdtran]

I'm not sure if this is what you're looking for, but here's the json of an event we recently got an email for.

{
  "version": "1.0",
  "oldState": {
    "stateValue": "ALARM",
    "stateReason": "Threshold Crossed: 1 datapoint (3.5420000000000003) was less than or equal to the threshold (30.0).",
    "stateReasonData": {
      "version": "1.0",
      "queryDate": "2016-08-30T18:41:08.688+0000",
      "startDate": "2016-08-30T18:35:00.000+0000",
      "unit": "Percent",
      "statistic": "Average",
      "period": 60,
      "recentDatapoints": [
        3.5420000000000003
      ],
      "threshold": 30
    }
  },
  "newState": {
    "stateValue": "INSUFFICIENT_DATA",
    "stateReason": "Insufficient Data: 1 datapoint was unknown.",
    "stateReasonData": {
      "version": "1.0",
      "queryDate": "2016-08-30T18:42:08.690+0000",
      "unit": "Percent",
      "statistic": "Average",
      "period": 60,
      "recentDatapoints": [],
      "threshold": 30
    }
  }
}

Thanks again for looking into it! Let me know if/how I can be more helpful.

[kqdtran]

I'm still having trouble getting this to work. If it makes any difference, our instances and RDS db live inside a VPC. Do we need to specify the VPC configurations for the lambda function to make this work?

[creichert]

The problem I see so far is that the event you posted doesn't appear to be the full event (it doesn't even mention the metric):

Here's what a CloudWatch notification looks like in our test data:

{
  "Records": [
    {
      "EventSource": "aws:sns",
      "EventVersion": "1.0",
      "EventSubscriptionArn": "arn:aws:sns:us-west-2:123456789123:CloudWatchNotifications:00000000-0000-0000-0000-000000000000",
      "Sns": {
        "Type": "Notification",
        "MessageId": "00000000-0000-0000-0000-000000000000",
        "TopicArn": "arn:aws:sns:us-west-2:123456789123:CloudWatchNotifications",
        "Timestamp": "2016-08-11T07:24:05.959Z",
        "Subject": "ALARM: \"awsrds-app-High-DB-Connections\" in US West - Oregon",
        "Message": "{\"AlarmName\":\"awsrds-app-High-DB-Connections\",\"AlarmDescription\":null,\"AWSAccountId\":\"123456789123\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 datapoint (10.0) was greater than or equal to the threshold (10.0).\",\"StateChangeTime\":\"2016-07-24T22:05:19.737+0000\",\"Region\":\"US West - Oregon\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"DatabaseConnections\",\"Namespace\":\"AWS/RDS\",\"Statistic\":\"AVERAGE\",\"Unit\":null,\"Dimensions\":[{\"name\":\"DBInstanceIdentifier\",\"value\":\"app\"}],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanOrEqualToThreshold\",\"Threshold\":10.0}}",
        "MessageAttributes": {}
      }
    }
  ]
}

When I check out our teams Lambda logs in cloudwatch, I see events like this:

• Do you see something like that?
• Did you run make test for this project and did it successfully post messages to your channel?

[creichert]

To my knowledge, you don't need any special VPC settings. You might double check your IAM lambda_exec_role to ensure it can execute the function and has no other problems.