(aws-s3-notifications): Unable to validate the following destination configurations.

[oste]

trafficstars

What is the problem?

Error: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unable to validate the following destination configurations. See the details in CloudWatch Log Stream:

Cloudwatch Log Stream didn't seem to have further details.

Reproduction Steps

    const importedBucketFromName = s3.Bucket.fromBucketName(
      this,
      'imported-bucket-from-name',
      'bucket-name'
    );

    const importedLambdaFromArn = lambda.Function.fromFunctionArn(
      this,
      'external-lambda-from-arn',
      'lambda-arn'
    );

    importedBucketFromName.addEventNotification(
      s3.EventType.OBJECT_CREATED,
      new s3n.LambdaDestination(importedLambdaFromArn),
      { prefix: 'test/', suffix: '.yaml' }
    );

What did you expect to happen?

Bucket would have an event notification for the given config pointing to the lambda.

What actually happened?

Error shown was generated.

CDK CLI Version

2.2.0 (build 4f5c27c)

Framework Version

n/a

Node.js Version

16.11

OS

OSX

Language

Typescript

Language Version

TypeScript (4.4.4)

Other information

From what I have read this seems to be caused by a missing policy on either the S3 Bucket or Lambda function or both. My attempts to add these policies have not worked.

Perhaps this issue stems from both resources(lambda, bucket) being imported, and thus addEventNotification is not able to add the required policies. Any workaround would be greatly apprecaited.

[NGL321]

Hey @oste,

Thank you for reporting this! I believe that this may have to do with the way you import your lambda function. Historically we have had trouble with imports by Arn alone as you can see here and here.

Have you tried importing with fromFunctionAttributes?

[oste]

hi @NGL321

Switching to fromFunctionAttributes produces the same Unable to validate the following destination configurations. error.

Here is what I changed to:

    const importedLambdaFromArn = lambda.Function.fromFunctionAttributes(
      this,
      'external-lambda-from-arn',
      {
        functionArn: 'my-arn',
      }
    );

[oste]

I even tried adding the role to the function attributes config, without success

        role: iam.Role.fromRoleArn(
          this,
          'role',
          'arn:aws:iam::123:role/myLambda'
        ),

[oste]

Hi @NGL321, I am trying to plan feature development around this issue. I am wondering if it's possible to get an idea of when this issue might be resolved. This way I can decide if it makes sense to look into a workaround.

Thanks in advance for any details.

[dandu1008]

@NGL321 Have any updates on this? It's really becoming a huge blocker for us. I have tried with fromFunctionAttributes alone and passing fromFunctionArn both. In either case, it is not working. Meantime, do you have any possible workaround to propose?

[YikaiHu]

We have met the same issue, but in SQS case. And we found this is an occasional issue.

We pointed the S3 event notification to an SQS with the following code,

// Add the S3 event on the log bucket with the target is sqs queue
logBucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.SqsDestination(logEventQueue), {
     prefix: props.logBucketPrefix
})

but got the same error "Error: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unable to validate the following destination configurations. See the details in CloudWatch Log Stream: "

And we had a look at the CloudWatch Log, but we cannot find some useful information. Here is the CloudWatch Log:

START RequestId: 41419846-addb-4806-b7dc-340c1ae7a2f9 Version: $LATEST
[ERROR] 2022-05-24T05:25:08.232Z 41419846-addb-4806-b7dc-340c1ae7a2f9 Failed to put bucket notification configuration
Traceback (most recent call last):
File "/var/task/index.py", line 28, in handler
put_bucket_notification_configuration(bucket, config)
File "/var/task/index.py", line 79, in put_bucket_notification_configuration
s3.put_bucket_notification_configuration(Bucket=bucket, NotificationConfiguration=notification_configuration)
File "/var/runtime/botocore/client.py", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 719, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unable to validate the following destination configurations
Status code: OK
END RequestId: 41419846-addb-4806-b7dc-340c1ae7a2f9
REPORT RequestId: 41419846-addb-4806-b7dc-340c1ae7a2f9 Duration: 1053.50 ms Billed Duration: 1054 ms Memory Size: 128 MB Max Memory Used: 73 MB Init Duration: 449.19 ms 
START RequestId: d78e4fd9-0e9a-4aba-8682-0ecc43969805 Version: $LATEST
Status code: OK
END RequestId: d78e4fd9-0e9a-4aba-8682-0ecc43969805
REPORT RequestId: d78e4fd9-0e9a-4aba-8682-0ecc43969805 Duration: 574.22 ms Billed Duration: 575 ms Memory Size: 128 MB Max Memory Used: 73 MB

And the workaround is delete the CloudFormation Stack and redeploy it.

[Adam-Randall]

I have also been coming across the same issue, my work around was to go and manually create it via console and then delete the same event notification. After doing this I was able to successfully deploy the event notification to AWS.

Not an ideal work around, but at least our CDK is not complaining.... Something behind the scenes with AWS must get created and not deleted following this.

[lili0allen]

hi @NGL321

Switching to fromFunctionAttributes produces the same Unable to validate the following destination configurations. error.

Here is what I changed to:

    const importedLambdaFromArn = lambda.Function.fromFunctionAttributes(
      this,
      'external-lambda-from-arn',
      {
        functionArn: 'my-arn',
      }
    );

I had the same issue. You need to set "sameEnvironment" flag to "true" in fromFunctionAttributes() to allow CDK has permission to update your external lambda function.

something like:

{ functionArn: 'my-arn', sameEnvironment: true }

[Kogs]

We have the same issue, and we create the bucket and the queue directly over the cdk script. Is there any solution for this? Sometimes it works most of the time not.

Edit: The Problem on our side was that the queue was encrypted and s3 did not have access to the kms key.

[lafeuil]

I have a similar issue when adding a event on a bucket that is managed by another account (cross account) to run a lambda. This error can appears when the account of the bucket is missing permissions for invoking the lambda.

According to AWS documentation, there are two types of permissions required:

• Permissions for your Lambda function to invoke services
• Permissions for Amazon S3 to invoke your Lambda function

I must add new resource-based policy because the current policy defined by the addEventNotification uses the current account in the source account property.

The current code is :

this.fn.addPermission(permissionId, {
  sourceAccount: Stack.of(bucket).account, // <- The issue is here with the account of the stack, not the account of the bucket 
  principal: new iam.ServicePrincipal('s3.amazonaws.com'),
  sourceArn: bucket.bucketArn,
  scope: bucket,
});

To fix the issue, add a new permission on the lambda with the correct account :

lambda.addPermission(permissionId, {
  sourceAccount: '12345678901234', // Owner Account id of the bucket 
  principal: new iam.ServicePrincipal('s3.amazonaws.com'),
  sourceArn: bucket.bucketArn,
});

I don't find any way to get the account id of a bucket.

[juissi-t]

We had the same issue. I contacted AWS support, and they were able to tell us that we had an existing event notification in the bucket for which the destination didn't exist anymore. We removed that notification and afterwards CDK could create the new event notification configuration correctly.

[wchaws]

We have met the same issue, but in SQS case. And we found this is an occasional issue.

We pointed the S3 event notification to an SQS with the following code,

// Add the S3 event on the log bucket with the target is sqs queue
logBucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.SqsDestination(logEventQueue), {
     prefix: props.logBucketPrefix
})

but got the same error "Error: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unable to validate the following destination configurations. See the details in CloudWatch Log Stream: "

And we had a look at the CloudWatch Log, but we cannot find some useful information. Here is the CloudWatch Log:

START RequestId: 41419846-addb-4806-b7dc-340c1ae7a2f9 Version: $LATEST
[ERROR] 2022-05-24T05:25:08.232Z 41419846-addb-4806-b7dc-340c1ae7a2f9 Failed to put bucket notification configuration
Traceback (most recent call last):
File "/var/task/index.py", line 28, in handler
put_bucket_notification_configuration(bucket, config)
File "/var/task/index.py", line 79, in put_bucket_notification_configuration
s3.put_bucket_notification_configuration(Bucket=bucket, NotificationConfiguration=notification_configuration)
File "/var/runtime/botocore/client.py", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 719, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unable to validate the following destination configurations
Status code: OK
END RequestId: 41419846-addb-4806-b7dc-340c1ae7a2f9
REPORT RequestId: 41419846-addb-4806-b7dc-340c1ae7a2f9 Duration: 1053.50 ms Billed Duration: 1054 ms Memory Size: 128 MB Max Memory Used: 73 MB Init Duration: 449.19 ms 
START RequestId: d78e4fd9-0e9a-4aba-8682-0ecc43969805 Version: $LATEST
Status code: OK
END RequestId: d78e4fd9-0e9a-4aba-8682-0ecc43969805
REPORT RequestId: d78e4fd9-0e9a-4aba-8682-0ecc43969805 Duration: 574.22 ms Billed Duration: 575 ms Memory Size: 128 MB Max Memory Used: 73 MB

And the workaround is delete the CloudFormation Stack and redeploy it.

By manually adding boto3.set_stream_logger(\"botocore\", logging.DEBUG) to the custom resource lambda generated by cdk for BucketNotifications, I found that the root cause of the error was due to insufficient permissions on the destination queue, preventing S3 from publishing notifications from the bucket. Given that the sqs access policy may take some time to propagate, the best solution would be to incorporate retry logic into the put bucket notification custom resource lambda.

The following is the boto3 debug log

b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InvalidArgument</Code><Message>Unable to validate the following destination configurations</Message><ArgumentName1>arn:aws:sqs:eu-north-1:*****:tCaT-clo-cognito-571690-CLFlbConfUploadingEventQueueAB242C04-ittCZ64K0mAg</ArgumentName1><ArgumentValue1>Permissions on the destination queue do not allow S3 to publish notifications from this bucket</ArgumentValue1><RequestId>SMY8RNEEN1PY4QB9</RequestId><HostId>YxuzvmGghxa3uVJ/o3OPB2wshteBI+1oWQWxSqY5g0ZKm5aLeW/tHBSIDXWiw8sOafq7NdD/BaI=</HostId></Error>'

The following is how I manually edit the generated template.

    "BucketNotificationsHandler050a0587b7544547bf325f094a3db8347ECC3691": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Description": "AWS CloudFormation handler for \"Custom::S3BucketNotifications\" resources (@aws-cdk/aws-s3)",
        "Code": {
          "ZipFile": "import boto3  # type: ignore\nimport json\nimport logging\nimport urllib.request\nboto3.set_stream_logger(\"botocore\", logging.DEBUG)\n\n ...

[manmartgarc]

We had the same issue. I contacted AWS support, and they were able to tell us that we had an existing event notification in the bucket for which the destination didn't exist anymore. We removed that notification and afterwards CDK could create the new event notification configuration correctly.

Gosh, can't thank you enough for sharing. I had to recreate the custom resource, in this case a Lambda function, reassign the resource-base policy and then I was finally able to delete the notification rule.

[Exter-dg]

I am facing the same issue when adding a new SNS event notification to my already-created S3 bucket. The bucket is created using CDK only.

Error - An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation

let bucket = new s3.Bucket(this, "Bucket", {
  bucketName: bucketName
}

// new code
const snsTopic = sns.Topic.fromTopicArn(
  this,
  `snsTopic`,
  "arn"
);

bucket.addEventNotification(
  s3.EventType.OBJECT_CREATED
  new s3n.SnsDestination(snsTopic)
);

Is there a workaround?

Similar to #28915

[GavinZZ]

@oste I notice that you're using imported lambda function. For imported function, we will set the property canCreatePermission to false, see code.

Since this property is set to false, then when binding the LambdaDestination to s3 event notifications, it will attempt to create lambda permisison (see code) but failed to create since canCreatePermissions is false.

Without necessary permission, creating a s3 event notification will then fail due to Unable to validate the following destination configurations

[Jerome3590]

This fixed it for me: https://sagargiri.com/s3-event-notification-issue

$ aws lambda add-permission
--function-name TestFunc:dev
--profile default
--statement-id AllowToBeInvoked
--action "lambda:InvokeFunction"
--principal s3.amazonaws.com
--source-arn "arn:aws:s3:::MyAwesomeBucket"
--source-account 123456789101

[alexiswl]

Rather than directly forward on events from the event queue, our workaround for this was to turn on event notifications on the bucket and then create rules on the default event bus that matched said buckets. We then add the sqs queue as a target to this rule.

This code snippet may be of use - https://github.com/umccr/orcabus/blob/eecbeaeef4ff20b588b9a7410bea9bac5a4a7919/lib/workload/stateful/stacks/shared/constructs/event-source/index.ts#L61-L100

[barreeeiroo]

I just faced this issue with a S3 > SQS integration. In my case, the reason for this "badly reported" failure was the KMS encryption in the SQS.

TL;DR: Verify if your SQS is using KMS. If it's using KMS, either allow S3 to decrypt the key, or change to the standard SSE-SQS encryption.

---

As additional details, in case anyone faces this, there is an easy way to troubleshoot this failure and get a proper fail reason without contacting support:

1. Open the S3 console of the affected bucket, and fill the details to try to create the notification manually.
2. Before hitting create, open Dev Tools and go to the Network tab.
3. Now hit Create, and see the failed request. This failed request is an actual request to the Control Plane XML API, and the response contains a detailed error message, indicating the exact fail reason.
4. If the failure is because of missing permission for S3 to send the message to SQS, you will see a message indicating that S3 lacks those permissions. Just go to SQS and add the resource policy.
5. If the resource policy is there, and your issue is related to KMS, then you will see some KMS-related error message in the failed request, mentioning something like S3 cannot access the key or use it.

Note that the addEventNotification methods in CDK will create the resource policy in SQS automatically. If CDK fails create the notifications, it will roll back that resource policy (that's why I have mentioned step 4 above). As such, for debugging it from the console, you may have to manually create the resource policy to "bypass" that error and get to the real deal.