using disableLogs option with a single httpApi fails to provision IamS Role

[calumbrodie]

When using the 'disableLogs' option with a function set up to use httpApi rather than the default 'restApi' (example configuration below), the resulting IAMS role is not created properly. The configuration is generated without any 'Resources'.

A workaround is to define a second 'fake' function which does not have 'disableLogs' option. This allows the role to be correctly generated.

The bug was likely introduced here: https://github.com/serverless/serverless/pull/8561/files

As you can see by looking at the code, it sort of relies on at least one defined function being able to get past the 'return' here: https://github.com/serverless/serverless/pull/8561/files#diff-634c192175fa19e078211273e8cec92851e3a2fb117fdf447b6998fa1af1a5cfR104

So if at least 1 function is not set up to 'log' then the policy is attempted to be created with an invalid 'Resource []' node (it should never be empty)

e.g

"PolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": [
                    "logs:CreateLogStream",
                    "logs:CreateLogGroup"
                  ],
                  "Resource": []
                },
                {
                  "Effect": "Allow",
                  "Action": [
                    "logs:PutLogEvents"
                  ],
                  "Resource": []
                }
              ]
            }
          }

I guess the next step would be to convert my provided serverless.yml into a failing test case?

serverless.yml

service: my-service
unresolvedVariablesNotificationMode: error
variablesResolutionMode: 20210219

provider:
  name: aws
  runtime: nodejs12.x
  region: ${opt:region}
  deploymentBucket:
    name: ssr-lambda-${opt:region}
  stage: ${opt:stage, self:custom.defaultStage}
  profile: ${self:custom.profiles.${self:provider.stage}}
  memorySize: 1024
  logRetentionInDays: 5
  lambdaHashingVersion: 20201221
  endpointType: REGIONAL
  apiGateway:
    shouldStartNameWithService: true

package:
  defaultStage: dev
  profiles:
    dev: devProfile

functions:
  ssr:
    handler: ./dist/index.handler
    disableLogs: true
    events:
      - httpApi: '*'

NODE_ENV=production webpack && sls deploy output

Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service our-code.zip file to S3 (13.22 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
....
Serverless: Operation failed!
Serverless: View the full error output: snipped

 Serverless Error ----------------------------------------

  An error occurred: IamRoleLambdaExecution - Policy statement must contain resources. (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: 29809a22-xxx-4988-a82d-xxxxxxx; Proxy: null).

Installed version

> NODE_ENV=production sls --version

Framework Core: 2.29.0 (local)
Plugin: 4.5.0
SDK: 4.2.0
Components: 3.7.3

[pgrzesik]

Hello @calumbrodie, thanks for reporting. I've managed to reproduce your issue locally. I believe in such situations we should not add the policy with logs:CreateLogStream, logs:CreateLogGroup and logs:PutLogEvents actions at all, it should ensure that the role can be created without issues.

We'd be happy to accept a PR that fixes that problem :+1:

[calumbrodie]

Hi,

I had a look and may be able to do something to fix but it would require a fairly large refactor of "mergeIamTemplates.js".

I think the general approach should be to first iterate the functions and store an array of named resources that need to be logged, and an array of ones that do not (disabled). Then depending on if these are empty or not then construct the resources appropriately. I think that would simplify the logic and make the code easier to follow.

What I couldn't understand from the above code was how functions which are 'CanonicallyNamed' need to be treated differently. It see from the code that 'CanonicallyNamed' functions start with the name of the service, from my example above 'my-service' but I'm not sure why that is important, why someone would name functions this way, and why it would have any effect on logging them?

[pgrzesik]

Hello @calumbrodie - that would be awesome :raised_hands:

I think the general approach should be to first iterate the functions and store an array of named resources that need to be logged, and an array of ones that do not (disabled). Then depending on if these are empty or not then construct the resources appropriately. I think that would simplify the logic and make the code easier to follow.

Could you elaborate a little bit more on that approach? Please keep in mind that during compileFunctions, there might be additional policies added caused by e.g. use of specific events.

What I couldn't understand from the above code was how functions which are 'CanonicallyNamed' need to be treated differently. It see from the code that 'CanonicallyNamed' functions start with the name of the service, from my example above 'my-service' but I'm not sure why that is important, why someone would name functions this way, and why it would have any effect on logging them?

Functions that are "CanonicallyNamed" don't have name property specified in the config and their name is generated internally to have form {service}-{stage}-{functionName} where functionName is equal to implicit name from config - in your case it would be ssr. The reason that they're treated differently is the fact that for canonically named functions, we can "catch them all" with resource being defined as

{
  "Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/some service-dev*:*"
}

where some-service is service and dev is stage. For custom functions, we have to specify each one of them as a separate resource.

Below you can see serverless.yml and corresponding CF template where you can see how the generated IamRoleLambdaExectution looks like and how it includes only one "resource" for policy statements for canonically named functions.

serverless.yml

service: some-service

provider:
  name: aws
  runtime: nodejs12.x
  region: us-east-1

functions:
  customFunc:
    name: customFuncName
    handler: index.handler
    description: Testing mybucket uploads
  canonicalA:
    handler: index.handler
  canonicalB:
    handler: index.handler

cloudformation.json

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "The AWS CloudFormation template for this Serverless application",
  "Resources": {
    "ServerlessDeploymentBucket": {
      "Type": "AWS::S3::Bucket",
      "Properties": {
        "BucketEncryption": {
          "ServerSideEncryptionConfiguration": [
            {
              "ServerSideEncryptionByDefault": {
                "SSEAlgorithm": "AES256"
              }
            }
          ]
        }
      }
    },
    "ServerlessDeploymentBucketPolicy": {
      "Type": "AWS::S3::BucketPolicy",
      "Properties": {
        "Bucket": {
          "Ref": "ServerlessDeploymentBucket"
        },
        "PolicyDocument": {
          "Statement": [
            {
              "Action": "s3:*",
              "Effect": "Deny",
              "Principal": "*",
              "Resource": [
                {
                  "Fn::Join": [
                    "",
                    [
                      "arn:",
                      {
                        "Ref": "AWS::Partition"
                      },
                      ":s3:::",
                      {
                        "Ref": "ServerlessDeploymentBucket"
                      },
                      "/*"
                    ]
                  ]
                },
                {
                  "Fn::Join": [
                    "",
                    [
                      "arn:",
                      {
                        "Ref": "AWS::Partition"
                      },
                      ":s3:::",
                      {
                        "Ref": "ServerlessDeploymentBucket"
                      }
                    ]
                  ]
                }
              ],
              "Condition": {
                "Bool": {
                  "aws:SecureTransport": false
                }
              }
            }
          ]
        }
      }
    },
    "CustomFuncLogGroup": {
      "Type": "AWS::Logs::LogGroup",
      "Properties": {
        "LogGroupName": "/aws/lambda/customFuncName"
      }
    },
    "CanonicalALogGroup": {
      "Type": "AWS::Logs::LogGroup",
      "Properties": {
        "LogGroupName": "/aws/lambda/some-service-dev-canonicalA"
      }
    },
    "CanonicalBLogGroup": {
      "Type": "AWS::Logs::LogGroup",
      "Properties": {
        "LogGroupName": "/aws/lambda/some-service-dev-canonicalB"
      }
    },
    "IamRoleLambdaExecution": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": [
                  "lambda.amazonaws.com"
                ]
              },
              "Action": [
                "sts:AssumeRole"
              ]
            }
          ]
        },
        "Policies": [
          {
            "PolicyName": {
              "Fn::Join": [
                "-",
                [
                  "some-service",
                  "dev",
                  "lambda"
                ]
              ]
            },
            "PolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": [
                    "logs:CreateLogStream",
                    "logs:CreateLogGroup"
                  ],
                  "Resource": [
                    {
                      "Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/customFuncName:*"
                    },
                    {
                      "Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/some-service-dev*:*"
                    }
                  ]
                },
                {
                  "Effect": "Allow",
                  "Action": [
                    "logs:PutLogEvents"
                  ],
                  "Resource": [
                    {
                      "Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/customFuncName:*:*"
                    },
                    {
                      "Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/some-service-dev*:*:*"
                    }
                  ]
                }
              ]
            }
          }
        ],
        "Path": "/",
        "RoleName": {
          "Fn::Join": [
            "-",
            [
              "some-service",
              "dev",
              {
                "Ref": "AWS::Region"
              },
              "lambdaRole"
            ]
          ]
        }
      }
    },
    "CustomFuncLambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Code": {
          "S3Bucket": {
            "Ref": "ServerlessDeploymentBucket"
          },
          "S3Key": "serverless/some-service/dev/1615989970804-2021-03-17T14:06:10.804Z/some-service.zip"
        },
        "Handler": "index.handler",
        "Runtime": "nodejs12.x",
        "FunctionName": "customFuncName",
        "MemorySize": 1024,
        "Timeout": 6,
        "Description": "Testing mybucket uploads",
        "Role": {
          "Fn::GetAtt": [
            "IamRoleLambdaExecution",
            "Arn"
          ]
        }
      },
      "DependsOn": [
        "CustomFuncLogGroup"
      ]
    },
    "CanonicalALambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Code": {
          "S3Bucket": {
            "Ref": "ServerlessDeploymentBucket"
          },
          "S3Key": "serverless/some-service/dev/1615989970804-2021-03-17T14:06:10.804Z/some-service.zip"
        },
        "Handler": "index.handler",
        "Runtime": "nodejs12.x",
        "FunctionName": "some-service-dev-canonicalA",
        "MemorySize": 1024,
        "Timeout": 6,
        "Role": {
          "Fn::GetAtt": [
            "IamRoleLambdaExecution",
            "Arn"
          ]
        }
      },
      "DependsOn": [
        "CanonicalALogGroup"
      ]
    },
    "CanonicalBLambdaFunction": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Code": {
          "S3Bucket": {
            "Ref": "ServerlessDeploymentBucket"
          },
          "S3Key": "serverless/some-service/dev/1615989970804-2021-03-17T14:06:10.804Z/some-service.zip"
        },
        "Handler": "index.handler",
        "Runtime": "nodejs12.x",
        "FunctionName": "some-service-dev-canonicalB",
        "MemorySize": 1024,
        "Timeout": 6,
        "Role": {
          "Fn::GetAtt": [
            "IamRoleLambdaExecution",
            "Arn"
          ]
        }
      },
      "DependsOn": [
        "CanonicalBLogGroup"
      ]
    },
    "CustomFuncLambdaVersionYVQZ7SLgjGZKtjB6M9P0rsNu0bd5a8V8THvaOWUxsA": {
      "Type": "AWS::Lambda::Version",
      "DeletionPolicy": "Retain",
      "Properties": {
        "FunctionName": {
          "Ref": "CustomFuncLambdaFunction"
        },
        "CodeSha256": "hxRv1eJvNlLRFpGqAFeJ/k9DGtCFOHXDSkhXYeavnig=",
        "Description": "Testing mybucket uploads"
      }
    },
    "CanonicalBLambdaVersiono0B0XczMXv07NRiCI4QT2iDB0ovCWqvM0IDed1HDo": {
      "Type": "AWS::Lambda::Version",
      "DeletionPolicy": "Retain",
      "Properties": {
        "FunctionName": {
          "Ref": "CanonicalBLambdaFunction"
        },
        "CodeSha256": "hxRv1eJvNlLRFpGqAFeJ/k9DGtCFOHXDSkhXYeavnig="
      }
    },
    "CanonicalALambdaVersionbuFxmWKXmKkVMXBdxdQfq9WJzwGJ0tCx8vP9cynh3OQ": {
      "Type": "AWS::Lambda::Version",
      "DeletionPolicy": "Retain",
      "Properties": {
        "FunctionName": {
          "Ref": "CanonicalALambdaFunction"
        },
        "CodeSha256": "hxRv1eJvNlLRFpGqAFeJ/k9DGtCFOHXDSkhXYeavnig="
      }
    }
  },
  "Outputs": {
    "ServerlessDeploymentBucketName": {
      "Value": {
        "Ref": "ServerlessDeploymentBucket"
      }
    },
    "CustomFuncLambdaFunctionQualifiedArn": {
      "Description": "Current Lambda function version",
      "Value": {
        "Ref": "CustomFuncLambdaVersionYVQZ7SLgjGZKtjB6M9P0rsNu0bd5a8V8THvaOWUxsA"
      }
    },
    "CanonicalBLambdaFunctionQualifiedArn": {
      "Description": "Current Lambda function version",
      "Value": {
        "Ref": "CanonicalBLambdaVersiono0B0XczMXv07NRiCI4QT2iDB0ovCWqvM0IDed1HDo"
      }
    },
    "CanonicalALambdaFunctionQualifiedArn": {
      "Description": "Current Lambda function version",
      "Value": {
        "Ref": "CanonicalALambdaVersionbuFxmWKXmKkVMXBdxdQfq9WJzwGJ0tCx8vP9cynh3OQ"
      }
    }
  }
}

Hope that clarifies the whole thing a bit - please let me know if I can provide some extra information :100:

[AnishLushte07]

Hi, Can someone guide me on disableLogs issue. I'm still facing it.

[Shereef]

ServerlessError: An error occurred: IamRoleLambdaExecution - Policy statement must contain resources. (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: cacab1ed-4684-48ad-b150-db73fd031d37; Proxy: null).
593 | at /codebuild/output/src687836509/src/service/node_modules/serverless/lib/plugins/aws/lib/monitor-stack.js:149:23
594 | at processTicksAndRejections (internal/process/task_queues.js:95:5)
595 | at AwsDeploy.createFallback (/codebuild/output/src687836509/src/service/node_modules/serverless/lib/plugins/aws/lib/update-stack.js:62:5) {
596 | code: 'AWS_CLOUD_FORMATION_CREATE_STACK_INTERNAL_I_A_M_ROLE_CREATE_FAILED',
597 | decoratedMessage: 'CREATE_FAILED: IamRoleLambdaExecution (AWS::IAM::Role)\n' +
598 | 'Policy statement must contain resources. (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: cacab1ed-4684-48ad-b150-db73fd031d37; Proxy: null)\n' +
599 | '\n' +
600

Any update on this issue ?

[drobakowski]

Hi there, any news on this? We're also getting this error while using a simple handler for SQS events:

...
SimpleConsumer:
  handler: foo.bar.SimpleConsumer
  disableLogs: true
  timeout: 45
  events:
    - sqs:
        arn: !GetAtt SimpleQueue.Arn
  vpc:
    ${self:custom.vpcSettings}
...

> sls deploy -s dev
✖ Stack simple-be-dev failed to deploy (37s)
Environment: darwin, node 18.3.0, framework 3.18.2 (local) 3.8.0v (global), plugin 6.2.2, SDK 4.3.2
Credentials: Local, environment variables
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
UPDATE_FAILED: IamRoleLambdaExecution (AWS::IAM::Role)
Policy statement must contain resources. (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: d569bf61-7aff-4491-b16b-6a23303374f4; Proxy: null)

[luiznazari]

The issue that apparently solved this problem was closed almost one year ago but I'm still facing this error as well.

serverless.ts

import type { AWS } from '@serverless/typescript'

const serverlessConfiguration: AWS = {
  service: 'service-name',
  frameworkVersion: '3',
  plugins: [
    'serverless-esbuild',
    'serverless-offline',
    'serverless-offline-ssm',
    'serverless-plugin-datadog',
    'serverless-step-functions',
  ],
  // ...
  functions: {
    'function-name': {
      // ...
      disableLogs: true
    }
  }
  // ...
}

$ node --version v14.18.1

serverless: v3.17.0 serverless-esbuild: v1.27.1 serverless-offline: v8.7.0 serverless-offline-ssm: v6.2.0 serverless-plugin-datadog: v5.1.1 serverless-plugin-typescript: v2.1.2 serverless-step-functions: v3.7.0

Temporary solution

I'm not proud of it but I managed to make a workaround for this issue.

As the CloudFormation template is generated with wrong "Resource" values and it's a JSON, I created a script that runs between sls package and sls deploy actions modifying the generated JSON CF Template removing those invalid resoures.

To do this:

1. I installed the serverless-plugin-scripts (v1.0.2) plugin;
2. Added this configuration to serverless.ts:

custom: {
    scripts: {
      hooks: {
        'package:finalize': 'node ./infra/scripts/fix_disableLogs_resource.js',
      },
    },
}

3. Created a script to modify the generated .serverless/serverless-state.json file, used by sls deploy.

fix_disableLogs_resource.js

const fs = require('fs')
const path = require('path')

// =-=-=-=-=: Functions :=-=-=-=-=

function readJsonFile(fileName) {
  const cfStackTemplateFile = path.resolve(`.serverless/${fileName}`)

  return {
    path: cfStackTemplateFile,
    content: require(cfStackTemplateFile),
  }
}

function backupFile(file) {
  const bkpPath = `${file.path}.bkp`
  if (!fs.existsSync(bkpPath)) {
    fs.copyFileSync(file.path, bkpPath)
  }
}

function hasResource(statement) {
  return !!statement.Resource?.length || !!statement.$ref
}

function fixLambdaExecutionPolicies(template) {
  const iamLambdaExecutionResource = template.Resources.IamRoleLambdaExecution
  if (iamLambdaExecutionResource) {
    const iamRolePolicies = iamLambdaExecutionResource.Properties.Policies

    iamRolePolicies.forEach(policy => {
      const statements = policy.PolicyDocument.Statement
      const statementWithValidResource = statements.filter(hasResource)
      policy.PolicyDocument.Statement = statementWithValidResource
    })
  }
}

function writeFile(file) {
  fs.writeFileSync(file.path, JSON.stringify(file.content))
}

// =-=-=-=-=: Execution :=-=-=-=-=

console.log(
  "Updating compiledCloudFormationTemplate excluding IAM Role's policy empty resources...",
)

// Only for consistency's sake
const stackFile = readJsonFile('cloudformation-template-update-stack.json')
backupFile(stackFile)
fixLambdaExecutionPolicies(stackFile.content)
writeFile(stackFile)

// This changes the real deployed package
const stateFile = readJsonFile('serverless-state.json')
backupFile(stateFile)
fixLambdaExecutionPolicies(
  stateFile.content.service.provider.compiledCloudFormationTemplate,
)
writeFile(stateFile)

console.log('compiledCloudFormationTemplate template updated.')

This is not perfect, but works and solved my problem.

[wzhonggo]

Simple case is work if use @luiznazari plan. But deploy fail if you use sqs event with Join function. You should use arn to fix this error.

serverless.yml

frameworkVersion: '3' 

plugins:
  - serverless-bundle
  - serverless-iamroles
  - serverless-plugin-scripts

custom:
  serverless-iamroles:
    defaultInherit: true
  scripts:
    hooks:
      'package:finalize': 'node ./../../scripts/fix_disableLogs_resource.js'


functions:
  demo:
    handler: demo.main
    disableLogs: true
    events:
      - sqs:
# use sqs arn     
#          arn: ${SQS_ARN}      
          arn:
            Fn::Join:
              - ':'
              - - arn
                - aws
                - sqs
                - Ref: AWS::Region
                - Ref: AWS::AccountId
                - ${SQS_Name}

error message

[luiznazari]

@wzhonggo

Today I faced the same issue with a SQS Event using my script. The only difference is that, instead of using JoinFunction, I was using GetAtt:

events: [
    {
      sqs: {
        arn: {
          'Fn::GetAtt': ['SqsRandomName', 'Arn'],
        },
      },
    },
  ],

The custom script modify the Iam policy statements array directly, it looks like any Serverless script referencing to a path to that object will fail as the array indexes changes.

The solution was to refere the SQS ARN as a string as you pointed. Thanks for that.

events: [
    {
      sqs: {
        arn: 'arn:aws:sqs:${aws:region}:${aws:accountId}:sqs-random-name'
      },
    },
  ],

[11-nth]

Hello I'm still having the same issue.

Deploying aws-python to stage dev (ap-southeast-1)

× Stack aws-python-dev failed to deploy (22s)
Environment: win32, node 18.13.0, framework 3.27.0, plugin 6.2.3, SDK 4.3.2
Credentials: Local, "default" profile                                                                                                                           
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
CREATE_FAILED: IamRoleLambdaExecution (AWS::IAM::Role)
Policy statement must contain resources. (Service: AmazonIdentityManagement; Status Code: 400; Error Code: 
MalformedPolicyDocument; Request ID: 27fe95eb-96ef-436f-81f5-1491264b1810; Proxy: null)

My serverless.yml only consists of:

service: aws-python

frameworkVersion: '3'

provider:
  name: aws
  runtime: python3.8
  region: ap-southeast-1

functions:
  hello:
    handler: handler.hello
    url: true
    events:
      - httpApi:
          path: /hello
          method: get
    disableLogs: true

[Justin1002]

Any updates on this?

[luksfarris]

This is a really interesting issue, that I've been seeing since at least one year (serverless 3.21.0). Today I decided to spend a little more time on it. My case is the following: deploy a function to AWS with disableLogs: true fails. I'm not willing to go the custom script route, and looking at the closed PR discussion, I don't think I can fix this myself. Here's what I tried:

• Removing the whole cloud formation stack and redeploying seems to work
  • I assume the same for commenting the functions, deploying, then uncommenting them and deploying
  • Deploying at least one function with disableLogs: false works
• Renaming the functions doesn't work
• Adding an overriding iamRoleStatements rule with effect Allow or Deny doesn't work
• Adding "custom" names (instead of the "canonical" name) with the name attribute doesn't work
• Updating to the latest sls version (3.33.0) doesn't work
• Adding a dummy function with logs enabled, and then removing it doesn't work.

So for now, I will have to always keep one of the functions with logs, unfortunately.

[osocode]

So far we have decided to disableLogs on everything but one function that doesn't emit many in order to stop paying for duplicate logging costs between CloudWatch and a third party we emit logs to directly from Lambda.

I'm really disappointed this feature has been broken for years and no hope in sight for a fix.