amazon-redshift-monitoring icon indicating copy to clipboard operation
amazon-redshift-monitoring copied to clipboard

Lambda function timing out while publishing cloudwatch metrics

Open bahugashish opened this issue 6 years ago • 1 comments

I deployed CloudWatch using SAM and I can see the data being fetched from Redshift Cluster within 1 minute, however the Lambda function still doesn't complete after running for 5 minutes,.

Lambda logs report: Executing Redshift Diagnostic Query: WLMQuerySlotCountWarning Publishing 24 CloudWatch Metrics END RequestId: b054f912-b614-11e8-aa9e-d5eb851e7827 REPORT RequestId: b054f912-b614-11e8-aa9e-d5eb851e7827 Duration: 300005.52 ms Billed Duration: 300000 ms Memory Size: 192 MB Max Memory Used: 32 MB 2018-09-11T22:52:41.248Z b054f912-b614-11e8-aa9e-d5eb851e7827 Task timed out after 300.01 seconds

Cloudwatch Log report:

 22:47:42 Executing Redshift Diagnostic Query: WLMQuerySlotCountWarning  22:47:42 Publishing 24 CloudWatch Metrics Publishing 24 CloudWatch Metrics  22:52:41 END RequestId: b054f912-b614-11e8-aa9e-d5eb851e7827 END RequestId: b054f912-b614-11e8-aa9e-d5eb851e7827  22:52:41 REPORT RequestId: b054f912-b614-11e8-aa9e-d5eb851e7827 Duration: 300005.52 ms Billed Duration: 300000 ms Memory Size: 192 MB Max Memory Used: 32 MB  22:52:41 2018-09-11T22:52:41.248Z b054f912-b614-11e8-aa9e-d5eb851e7827 Task timed out after 300.01 seconds  22:52:41 Pushing metrics to CloudWatch failed: exception ('Connection aborted.', error(1, 'Operation not permitted'))  22:52:41 /var/task/redshift_monitoring.py:249: SyntaxWarning: name 'debug' is assigned to before global declaration  22:52:41 global debug

bahugashish avatar Sep 11 '18 22:09 bahugashish

The function hang at below statement:

    try:
        cw.put_metric_data(
            Namespace='Redshift',
            MetricData=put)

Verified the IAM Role has put permissions as suggested, also tried giving full access to the Role and it did not help. Please suggest.

bahugashish avatar Sep 12 '18 00:09 bahugashish