aws-lambda-ses-forwarder
aws-lambda-ses-forwarder copied to clipboard
arithmetric
'copyObject() returned error:'
I have followed exactly what you've spelt out on https://github.com/arithmetric/aws-lambda-ses-forwarder but am stuck. No amount of Google search or No playing with options either has helped. A client is about to terminate the contract - we've stuck at this error for too long.
I have verified my domain. I receive email from anywhere through my verified address to my designated S3 bucket. I also changed the '"functionArn":xxx..."' to '"functionArn": "arn:aws:lambda:us-west-2:MY-ACCOUNT-ID:function:fnForwardEmailToGmail"'.
However the received emails are never forwarded and when I test the function using "SES Email Receiving" I get the error below.
START RequestId: 7f2cd5ed-83ea-11e7-913f-55748388c69f Version: $LATEST
2017-08-18T07:54:44.467Z 7f2cd5ed-83ea-11e7-913f-55748388c69f { level: 'info',
message: 'Fetching email at s3://MY-S3-BUCKET/MY-PREFIX/o3vrnil0e2ic28fgfdstrm7dfhrc2v0clambda4nbp0g1' }
2017-08-18T07:54:46.068Z 7f2cd5ed-83ea-11e7-913f-55748388c69f { level: 'error',
message: 'copyObject() returned error:',
error:
{ [AccessDenied: Access Denied]
message: 'Access Denied',
code: 'AccessDenied',
region: null,
time: Fri Aug 18 2017 07:54:46 GMT+0000 (UTC),
requestId: 'A6285517D1AF2B9D',
extendedRequestId: 'dfH3csS5kHLsYN4ZgIWVliYmuVb1OgCVl6KdUSdZdqwX2T+JdkfZwIyPa5KEgYFiJfZmrwXjXDI=',
cfId: undefined,
statusCode: 403,
retryable: false,
retryDelay: 32.49475641641766 },
stack: 'AccessDenied: Access Denied
at Request.extractError (/var/task/node_modules/aws-sdk/lib/services/s3.js:473:35)
at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:615:14)
at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.
LAMBDA ROLE FUNCTION POLICY { "Version": "2016-03-04", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": "arn:aws:logs:::" }, { "Effect": "Allow", "Action": "ses:SendRawEmail", "Resource": "" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject" ], "Resource": "arn:aws:s3:::MY-S3-BUCKET/*" } ] }
S3 BUCKET POLICY { "Version": "2012-10-17", "Statement": [ { "Sid": "GiveSESPermissionToWriteEmail", "Effect": "Allow", "Principal": { "Service": "ses.amazonaws.com" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::MY-S3-BUCKET/*", "Condition": { "StringEquals": { "aws:Referer": "MY-ACCOUNT-ID" } } } ] }
You probably used Policy template as is without change variable names to your values. That's mean, MY-S3-BUCKET must be replaced by name of S3 bucket you created before, MY-ACCOUNT-ID must be replaced by ID of your account.
As I said, Set Up guide contanins only templates of Policy. You must create your own S3 bucket and their name put info Policy. Please try to repeat (or check) all steps again and very carefully.
I am getting this error too, despite I populated this template with my values
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GiveSESPermissionToWriteEmail",
"Effect": "Allow",
"Principal": {
"Service": "ses.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::S3-BUCKET-NAME/*",
"Condition": {
"StringEquals": {
"aws:Referer": "AWS-ACCOUNT-ID"
}
}
}
]
}
Here is the detailed text error:
2017-09-22T10:16:53.807Z 27a1af58-9f7f-11e7-9f03-7f5301db9fb9 { level: 'error',
message: 'copyObject() returned error:',
error:
{ [AccessDenied: Access Denied]
message: 'Access Denied',
code: 'AccessDenied',
region: null,
time: Fri Sep 22 2017 10:16:53 GMT+0000 (UTC),
requestId: 'A3D512C38522FA3E',
extendedRequestId: '4ki2/g7CK5gukT5IIDLHb4hNsXsrG0/vwRpdtVXHqV5E3st4PLw/mEMzONQq51N1VLqQyFyb6Ew=',
cfId: undefined,
statusCode: 403,
retryable: false,
retryDelay: 28.15239334013313 },
stack: 'AccessDenied: Access Denied\n at Request.extractError (/var/task/node_modules/aws-sdk/lib/services/s3.js:473:35)\n at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:105:20)\n at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:77:10)\n at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:615:14)\n at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)\n at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:617:12)\n at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:115:18)' }
Could be region: null the reason?
This policy from S3 bucket you paste here is not applied for copyObject() method in SES.
You need check Role's policy for Lambda. In console for Lambda open your Lambda function, switch to Configuration tab and get name of assigned role in Existing role field. Then go to Roles section, open the Role from Lambda and check policy in Pesmisions tab if is according to template from Set Up Guide (point 2.)
Here is my role policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": "ses:SendRawEmail",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::*"
}
]
}
What is wrong with it?
Last block shoud look like:
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::your-bucket-name/*"
}
I tried both ways, but had the same result. Here in troubleshooting section they say that broader permissions can be a solution for AccessDenied: Access Denied error.
Maybe I should reload policy in some way? But I suppose, it is auto-reloadable.
Please put here previous line from log: The row contain text: Fetching email at…
Here it is:
11:19:23 2017-09-22T11:19:23.129Z 73f9eecf-9f87-11e7-8e69-4744598dde16 { level: 'info', message: 'Fetching email at s3://mailingo/undefined9d8l2udqruoak5sblo038l8mo0gk6pf3j7rmem81' }
11:19:23 2017-09-22T11:19:23.919Z 73f9eecf-9f87-11e7-8e69-4744598dde16 { level: 'error', message: 'copyObject() returned error:', error: { [AccessDenied: Access Denied] message: 'Access Denied', code: 'AccessDenied', region: null, time: Fri Sep 22 2017 11:19:23 GMT+0000 (UTC), requestId: '79727D68EBD8F51E', extendedRequestId: 'S3TjmqZ3+NyUpjKIZuNp92gzW7M6sPqN8MjT3GRArA
11:19:23 2017-09-22T11:19:23.931Z 73f9eecf-9f87-11e7-8e69-4744598dde16 {"errorMessage":"Error: Could not make readable copy of email."}
11:19:23 END RequestId: 73f9eecf-9f87-11e7-8e69-4744598dde16
11:19:23 REPORT RequestId: 73f9eecf-9f87-11e7-8e69-4744598dde16 Duration: 906.71 ms Billed Duration: 1000 ms Memory Size: 128 MB Max Memory Used: 39 MB
See the undefined in s3://mailingo/undefined9d8l2udqruoak5sblo038l8mo0gk6pf3j7rmem81
That mean You probably deleted the emailKeyPrefix from function config object and now it can't build valid Key for Object and copyObject() failed.
Just put
emailKeyPrefix: "",
for no prefix.
Note: You may think the right error message should be NoSuchKey as linked post say. No, it isn't, because Lambda have not ListBucket permission in Rule.
Exactly! Now it works. Thanks for the assistance. In one of the manuals I read that I shouldn't use prefix if my messages are put into S3 root. I deleted this value pair, instead putting it blank :)
P.S. BTW, from where do you know my bucket name? I've never revealed it until you comment :). Do you have access to all aws-lambda-ses-forwarder Lambda instances?
I read name of your bucker on this Github issute before you change the Issue text. You sent it here first and change it later. And I have it in e-mail issue notification.
Nobody than you can't see your AWS objects, Lambda instances of this tool included of course :).
I seem to be having a similar issue:
var defaultConfig = {
fromEmail: "[email protected]",
subjectPrefix: "",
emailBucket: "s3-us-west-2.amazonaws.com/ses-bar",
emailKeyPrefix: "emails/",
forwardMapping: {
"[email protected]": [
"[email protected]"
]
}
Line before error message:
2018-02-27T19:54:53.286Z a9802609-1bf7-11e8-8e8c-4dceb5d4bfb0 { level: 'info',
message: 'Fetching email at s3://s3-us-west-2.amazonaws.com/ses-bar/emails/4food1j34v2fc85nqhfmbn3p8ehfva5hsjm7ug81' }
If I go to my bucket and download 4food1j34v2fc85nqhfmbn3p8ehfva5hsjm7ug81 I see the email that I sent.
Error:
2018-02-27T19:54:53.587Z a9802609-1bf7-11e8-8e8c-4dceb5d4bfb0 { level: 'error',
message: 'copyObject() returned error:',
error:
{ AccessDenied: Access Denied
Bucket policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowSESPuts",
"Effect": "Allow",
"Principal": {
"Service": "ses.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::ses-bar/*",
"Condition": {
"StringEquals": {
"aws:Referer": "{hasMyRealAccounNumberHere}"
}
}
}
]
}
LambdaSesForwarder policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GiveSESPermissionToInvokeFunction",
"Effect": "Allow",
"Action": [
"lambda:InvokeFunction"
],
"Resource": "arn:aws:lambda:us-west-2:248230402177:function:SesForwarder"
},
{
"Effect": "Allow",
"Action": "ses:SendRawEmail",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::ses-bar/*"
}
]
}
You need a role with the S3 permission policy with a trust relationship to the lambda service:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
I've written a Terraform module to set this up at: killTERM.xyz
on my LambdaSesForwarder policy under trust relationship:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
I already had that i think.
I had a similar issue
2018-03-21T05:00:38.304Z ca238edd-2cc4-11e8-b015-09735f371c0d { level: 'error', message: 'copyObject() returned error:', error: { [NoSuchKey: The specified key does not exist.] message: 'The specified key does not exist.', code: 'NoSuchKey', region: null,
but got it resolved by setting the emailKeyPrefix to "" and fixing my Lambda Role
emailKeyPrefix: ""
For Lambda role, followed exactly what jakubboucek commented above :
"You need check Role's policy for Lambda. In console for Lambda open your Lambda function, switch to Configuration tab and get name of assigned role in Existing role field. Then go to Roles section, open the Role from Lambda and check policy in Pesmisions tab if is according to template from Set Up Guide (point 2.)"
Thank you jakubboucek for this function.
-mS
Weird, but I have that problem again after changing the domain. Last time I was missing emailKeyPrefix: "", from my config, but this time all seems to be perfect.
Lambda role policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": "ses:SendRawEmail",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::themail/*"
}
]
}
Bucket policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GiveSESPermissionToWriteEmail",
"Effect": "Allow",
"Principal": {
"Service": "ses.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::themail/*",
"Condition": {
"StringEquals": {
"aws:Referer": "1231354654"
}
}
}
]
}
Config of Lambda:
var overrides = {
config: {
fromEmail: "[email protected]",
subjectPrefix: "",
emailBucket: themail",
emailKeyPrefix: "",
forwardMapping: {
"@themail.com": [
"[email protected]"
]
}
}
};
LambdaForwarder.handler(event, context, overrides);
};
Errors:
20:20:30
2018-10-09T20:20:30.116Z c3fb69e8-cc00-11e8-9e9b-d5d61d0c877f { level: 'info', message: 'Fetching email at s3:/themail/emailsPrefix/3e0gj0hlljk4e8og2m92k84ss81lj31rolcchtg1' }
2018-10-09T20:20:30.116Z c3fb69e8-cc00-11e8-9e9b-d5d61d0c877f { level: 'info',
message: 'Fetching email at s3://themail/emailsPrefix/3e0gj0hlljk4e8og2m92k84ss81lj31rolcchtg1' }
20:20:31
2018-10-09T20:20:31.116Z c3fb69e8-cc00-11e8-9e9b-d5d61d0c877f { level: 'error', message: 'copyObject() returned error:', error: { AccessDenied: Access Denied at Request.extractError (/var/task/node_modules/aws-sdk/lib/services/s3.js:473:35) at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:105:20) at Request.emit (/var/task/node_modules/aws-sdk
2018-10-09T20:20:31.116Z c3fb69e8-cc00-11e8-9e9b-d5d61d0c877f { level: 'error',
message: 'copyObject() returned error:',
error:
{ AccessDenied: Access Denied
at Request.extractError (/var/task/node_modules/aws-sdk/lib/services/s3.js:473:35)
at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:615:14)
at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:617:12)
at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
message: 'Access Denied',
code: 'AccessDenied',
region: null,
time: 2018-10-09T20:20:31.114Z,
requestId: 'C44A03F07AEC479D',
extendedRequestId: '1kHi64DSuEZeg7I3xxge6zma7XT0L1mSDz5Hu7W8tMmJuXtQd97KyFkUVx+2iaGgF5IsmevVjVs=',
cfId: undefined,
statusCode: 403,
retryable: false,
retryDelay: 71.21399525942986 },
stack: 'AccessDenied: Access Denied\n at Request.extractError (/var/task/node_modules/aws-sdk/lib/services/s3.js:473:35)\n at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:105:20)\n at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:77:10)\n at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:615:14)\n at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)\n at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:617:12)\n at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:115:18)' }
20:20:31
2018-10-09T20:20:31.154Z c3fb69e8-cc00-11e8-9e9b-d5d61d0c877f { level: 'error', message: 'Step returned error: Error: Could not make readable copy of email.', error: Error: Error: Could not make readable copy of email. at Response.<anonymous> (/var/task/index.js:154:11) at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:355:18) at Request.callListeners (/var/task
2018-10-09T20:20:31.154Z c3fb69e8-cc00-11e8-9e9b-d5d61d0c877f { level: 'error',
message: 'Step returned error: Error: Could not make readable copy of email.',
error: Error: Error: Could not make readable copy of email.
at Response.<anonymous> (/var/task/index.js:154:11)
at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:355:18)
at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:615:14)
at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:617:12),
stack: 'Error: Error: Could not make readable copy of email.\n at Response.<anonymous> (/var/task/index.js:154:11)\n at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:355:18)\n at Request.callListeners (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:105:20)\n at Request.emit (/var/task/node_modules/aws-sdk/lib/sequential_executor.js:77:10)\n at Request.emit (/var/task/node_modules/aws-sdk/lib/request.js:615:14)\n at Request.transition (/var/task/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/var/task/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /var/task/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:38:9)\n at Request.<anonymous> (/var/task/node_modules/aws-sdk/lib/request.js:617:12)' }
20:20:31
2018-10-09T20:20:31.155Z c3fb69e8-cc00-11e8-9e9b-d5d61d0c877f {"errorMessage":"Error: Step returned error.","errorType":"Error","stackTrace":["/var/task/index.js:320:28","<anonymous>","process._tickDomainCallback (internal/process/next_tick.js:228:7)"]}
2018-10-09T20:20:31.155Z c3fb69e8-cc00-11e8-9e9b-d5d61d0c877f
{
"errorMessage": "Error: Step returned error.",
"errorType": "Error",
"stackTrace": [
"/var/task/index.js:320:28",
"<anonymous>",
"process._tickDomainCallback (internal/process/next_tick.js:228:7)"
]
}
Any suggestions, @jakubboucek?
There is no opening " in your emailBucket. Maybe that is the problem.
var overrides = { config: { fromEmail: "[email protected]", subjectPrefix: "", emailBucket: themail", emailKeyPrefix: "", forwardMapping: { "@themail.com": [ "[email protected]" ] } } }; LambdaForwarder.handler(event, context, overrides); };
The issue is in bucket permissions. When I change config to old bucket, all goes fine from all addresses:
var defaultConfig = {
fromEmail: "[email protected]",
subjectPrefix: "",
emailBucket: "oldmail",
emailKeyPrefix: "",
forwardMapping: {
"@oldmail.com": [
"[email protected]"
],
"@themail.com": [
"[email protected]"
]
}
};
And policies of both buckets are absolutely identical:
themail
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GiveSESPermissionToWriteEmail",
"Effect": "Allow",
"Principal": {
"Service": "ses.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::themail/*",
"Condition": {
"StringEquals": {
"aws:Referer": "1231354654"
}
}
}
]
}
oldmail
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GiveSESPermissionToWriteEmail",
"Effect": "Allow",
"Principal": {
"Service": "ses.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::oldmail/*",
"Condition": {
"StringEquals": {
"aws:Referer": "1231354654"
}
}
}
]
}
I was fighting this same error all morning and determined my own error in provisioning the bucket with a custom KMS key that was causing the CopyObject() call to fail.
If you are setting encryption on the s3 bucket level using custom keys. Ensure your IAM role assigned to the Lambda function has the ability to use the custom key in addition to put/pull objects. To directly reference AWS documentation:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ses:SendRawEmail",
"Resource": "*"
},
{
"Sid": "AllowPutGetMail",
"Effect": "Allow",
"Action": ["s3:PutObject", "s3:GetObject"],
"Resource": [
"arn:aws:s3:::examplebucket/*"
]
},
{
"Sid": "DecryptS3Objects",
"Effect": "Allow",
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": [
"arn:aws:kms:example-region-1:123456789098:key/111aa2bb-333c-4d44-5555-a111bb2c33dd"
]
}
]
}
