serverless
serverless copied to clipboard
serverless
Maximum policy size of 10240 bytes exceeded with lots of schedule events
Are you certain it's a bug?
- [X] Yes, it looks like a bug
Is the issue caused by a plugin?
- [X] It is not a plugin issue
Are you using the latest version?
- [X] Yes, I'm using the latest version
Is there an existing issue for this?
- [X] I have searched existing issues, it hasn't been reported yet
Issue description
When adding multiple schedule events to one Lambda function, an individual permission statement is created for each schedule to allow it to invoke the Lambda function. This results in a large policy document that eventually exceeds AWS's 10KB limit. In my specific case, >=29 schedules were enough to cause the issue.
These individual permissions could be replaced with a single permission containing a list of resources. The same scenario has occurred in a couple of other areas in the past, and has been resolved with this method:
- https://github.com/serverless/serverless/issues/2508
- https://github.com/serverless/serverless/issues/4686
- https://github.com/serverless/serverless/issues/7643
Service configuration (serverless.yml) content
functions:
dataWarehouseExport:
handler: dataWarehouseExport
memorySize: 10240
layers:
- ${self:custom.runtimeLayer}
events:
- schedule:
rate: cron(1 3 * * ? *) # Daily, at 03:01
input:
command: app:data-warehouse:REDACTED0
- schedule:
rate: cron(2 3 * * ? *) # Daily, at 03:02
input:
command: app:data-warehouse:REDACTED1
- schedule:
rate: cron(3 3 * * ? *) # Daily, at 03:03
input:
command: app:data-warehouse:REDACTED2
- schedule:
rate: cron(4 3 * * ? *) # Daily, at 03:04
input:
command: app:data-warehouse:REDACTED3
- schedule:
rate: cron(6 3 * * ? *) # Daily, at 03:06
input:
command: app:data-warehouse:REDACTED4
- schedule:
rate: cron(7 3 * * ? *) # Daily, at 03:07
input:
command: app:data-warehouse:REDACTED5
- schedule:
rate: cron(5 3 * * ? *) # Daily, at 03:05
input:
command: app:data-warehouse:REDACTED6
- schedule:
rate: cron(8 3 * * ? *) # Daily, at 03:08
input:
command: app:data-warehouse:REDACTED7
- schedule:
rate: cron(9 3 * * ? *) # Daily, at 03:09
input:
command: app:data-warehouse:REDACTED8
- schedule:
rate: cron(10 3 * * ? *) # Daily, at 03:10
input:
command: app:data-warehouse:REDACTED9
- schedule:
rate: cron(11 3 * * ? *) # Daily, at 03:11
input:
command: app:data-warehouse:REDACTED10
- schedule:
rate: cron(12 3 * * ? *) # Daily, at 03:12
input:
command: app:data-warehouse:REDACTED11
- schedule:
rate: cron(13 3 * * ? *) # Daily, at 03:13
input:
command: app:data-warehouse:REDACTED12
- schedule:
rate: cron(14 3 * * ? *) # Daily, at 03:14
input:
command: app:data-warehouse:REDACTED13
- schedule:
rate: cron(15 3 * * ? *) # Daily, at 03:15
input:
command: app:data-warehouse:REDACTED14
- schedule:
rate: cron(16 3 * * ? *) # Daily, at 03:16
input:
command: app:data-warehouse:REDACTED15
- schedule:
rate: cron(17 3 * * ? *) # Daily, at 03:17
input:
command: app:data-warehouse:REDACTED16
- schedule:
rate: cron(18 3 * * ? *) # Daily, at 03:18
input:
command: app:data-warehouse:REDACTED17
- schedule:
rate: cron(19 3 * * ? *) # Daily, at 03:19
input:
command: app:data-warehouse:REDACTED18
- schedule:
rate: cron(20 3 * * ? *) # Daily, at 03:20
input:
command: app:data-warehouse:REDACTED19
- schedule:
rate: cron(21 3 * * ? *) # Daily, at 03:21
input:
command: app:data-warehouse:REDACTED20
- schedule:
rate: cron(22 3 * * ? *) # Daily, at 03:22
input:
command: app:data-warehouse:REDACTED21
- schedule:
rate: cron(23 3 * * ? *) # Daily, at 03:23
input:
command: app:data-warehouse:REDACTED22
- schedule:
rate: cron(24 3 * * ? *) # Daily, at 03:24
input:
command: app:data-warehouse:REDACTED23
- schedule:
rate: cron(25 3 * * ? *) # Daily, at 03:25
input:
command: app:data-warehouse:REDACTED24
- schedule:
rate: cron(26 3 * * ? *) # Daily, at 03:26
input:
command: app:data-warehouse:REDACTED25
- schedule:
rate: cron(27 3 * * ? *) # Daily, at 03:27
input:
command: app:data-warehouse:REDACTED26
- schedule:
rate: cron(43 3 * * ? *) # Daily, at 03:43
input:
command: app:data-warehouse:REDACTED27
- schedule:
rate: cron(46 3 * * ? *) # Daily, at 03:46
input:
command: app:data-warehouse:REDACTED28
- schedule:
rate: cron(29 3 * * ? *) # Daily, at 03:29
input:
command: app:data-warehouse:REDACTED29
- schedule:
rate: cron(28 3 * * ? *) # Daily, at 03:28
input:
command: app:data-warehouse:REDACTED30
- schedule:
rate: cron(30 3 * * ? *) # Daily, at 03:30
input:
command: app:data-warehouse:REDACTED31
- schedule:
rate: cron(0 4 * * ? *) # Daily, at 04:00
input:
command: app:data-warehouse:REDACTED32
- schedule:
rate: cron(31 3 * * ? *) # Daily, at 03:31
input:
command: app:data-warehouse:REDACTED33
- schedule:
rate: cron(32 3 * * ? *) # Daily, at 03:32
input:
command: app:data-warehouse:REDACTED34
- schedule:
rate: cron(33 3 * * ? *) # Daily, at 03:33
input:
command: app:data-warehouse:REDACTED35
- schedule:
rate: cron(34 3 * * ? *) # Daily, at 03:34
input:
command: app:data-warehouse:REDACTED36
- schedule:
rate: cron(35 3 * * ? *) # Daily, at 03:35
input:
command: app:data-warehouse:REDACTED37
- schedule:
rate: cron(36 3 * * ? *) # Daily, at 03:36
input:
command: app:data-warehouse:REDACTED38
- schedule:
rate: cron(44 3 * * ? *) # Daily, at 03:44
input:
command: app:data-warehouse:REDACTED39
- schedule:
rate: cron(41 3 * * ? *) # Daily, at 03:41
input:
command: app:data-warehouse:REDACTED40
- schedule:
rate: cron(37 3 * * ? *) # Daily, at 03:37
input:
command: app:data-warehouse:REDACTED41
- schedule:
rate: cron(42 3 * * ? *) # Daily, at 03:42
input:
command: app:data-warehouse:REDACTED42
- schedule:
rate: cron(38 3 * * ? *) # Daily, at 03:38
input:
command: app:data-warehouse:REDACTED43
- schedule:
rate: cron(39 3 * * ? *) # Daily, at 03:39
input:
command: app:data-warehouse:REDACTED44
- schedule:
rate: cron(40 3 * * ? *) # Daily, at 03:40
input:
command: app:data-warehouse:REDACTED45
- schedule:
rate: cron(45 3 * * ? *) # Daily, at 03:45
input:
command: app:data-warehouse:REDACTED46
Command name and used flags
sls deploy -s sandbox
Command output
…REDACTED…
Serverless Error ----------------------------------------
An error occurred: DataWarehouseExportLambdaPermissionEventsRuleSchedule29 - The final policy size (20764) is bigger than the limit (20480). (Service: AWSLambda; Status Code: 400; Error Code: PolicyLengthExceededException; Request ID: a9945e0b-d7c2-4bbd-b34c-e323bd82a3ff; Proxy: null).
Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com
Your Environment Information ---------------------------
Operating System: darwin
Node Version: 12.22.7
Framework Version: 2.66.2
Plugin Version: 5.5.1
SDK Version: 4.3.0
Components Version: 3.18.1
Environment information
Framework Core: 2.66.2
Plugin: 5.5.1
SDK: 4.3.0
Components: 3.18.1
@jaikdean great thanks for report. A PR that improves handling of that is welcome!
Hello @simonarcher99 - we'd love to accept a contribution for this issue 🙇
The SourceArn property in a resourced based policy cannot take a list of arns so I don't believe this can be solved the same way as the other linked PRs. AWS suggest wildcarding the arn. For the above example this would look like a source arn of ${service}-${stage}-DataWarehouseExportPermissionEventsRuleSchedule*. Is this an acceptable solution?
If the user chooses to name their schedules (instead of using the default serverless naming) I can't think of a method that would work.
Thanks a lot for the deep dive @simonarcher99 👍 I think the wildcard approach might be a good choice there, it would solve the problem at least for users that didn't opt to name their schedules.
@medikoo @mnapoli What do you think?
I think that wildcard is the way to go. It's already very specific to service, so there's no real risk of leaking permissions
@simonarcher99, any updates on this issue? I'm also having the same problem when adding lots of scheduled events. Thanks for handling it.
@simonarcher99 @pgrzesik @medikoo any progress on this issue? Having trouble scheduling events for a single lambda, and currently our workaround is duplicating the lambda and having them split the scheduled events. Feels super hacky :/
