Operator library lambdas are unintentionally added to the StepFunctionRole inline policies when MIE is deployed as a nested stack

[aburkleaux-amazon]

Describe the bug

Originally reported on Gitter: https://gitter.im/awslabs/aws-media-insights-engine#:~:text=Hey%20chaps%2C%20another,something%20like%20this%3A

After adding 3 custom operators to the operator stack, the workflow API custom resource returns the following error:

Exception An error occurred (LimitExceeded) when calling the PutRolePolicy operation: Maximum policy size of 10240 bytes exceeded for role stackMediaInsightsEngine-StepFunctionRole-PG...

Suspected cause:

The code to create a new operator will not add an inline policy to execute the lambda in the StepFunctionExecutionRole if it is already part of the MIE operator library. The create operator code checs for this by looking for the string "OperatorLibrary" in the lambda names. With MIE deployed as a nested stack as part of the Content Analysis Application stack, the lambda names for operator library lambdas look like the example below and do not contain the string "OperatorLibrary":

mieclor-MieStack-<SHORTUUID>-CheckMediaConvertFunctio-<UUID>

This causes the code block below to execute and unintentionally adds inline policies for the MIE Operator Library lambdas to the StepFunctionExecutionRole. Since there is a size limit to inline policies, the limit is reached much sooner than expected:

https://github.com/awslabs/aws-media-insights-engine/blob/fc48f7fcbed967316e80d5fb8ce0bb5be069f335/source/workflowapi/app.py#L454

To Reproduce

1. Deploy MIE as a nested stack in the same way as the Content Analysis application: https://github.com/awslabs/aws-content-analysis one-click deploy.
2. Add new custom operators to the stack until you see the error. (TBD - Need to find out how operators were added)

Expected behavior There should be a higher limit to the number of custom operators that can be added. Based on discussion in the code here ,

        # TODO: Once IAM supports the ability to use tag-based policies for
        #  InvokeFunction, put that in the StepFunctionRole definition in
        #  media-insights-stack.yaml and remove the following code block. Inline
        #  policies have length limitations which will prevent users from adding
        #  more than about 35 new operators via the MIE workflow api. Tag based
        #  policies will not have any such limitation.

Please complete the following information about the solution:

• [ ] Version: [e.g. v2.0.5]

To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0021) - Video On Demand workflow with AWS Step Functions, MediaConvert, MediaPackage, S3, CloudFront and DynamoDB. Version v5.0.0". If the description does not contain the version information, you can look at the mappings section of the template:

Mappings:
  SourceCode:
    General:
      S3Bucket: "solutions"
      KeyPrefix: "video-on-demand-on-aws/v5.0.0"

• [ ] Region: [e.g. us-east-1]
• [ ] Was the solution modified from the version published on this repository?
• [ ] If the answer to the previous question was yes, are the changes available on GitHub?
• [ ] Have you checked your service quotas for the sevices this solution uses?
• [ ] Were there any errors in the CloudWatch Logs?

Screenshots If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context

Related to https://github.com/awslabs/aws-media-insights-engine/issues/291

[aburkleaux-amazon]

Another variation of this with CDK from Gitter:

Jack Liddiard @deadcore 00:38 We're adding [custom operators] through custom resources in CDK and the provided custom resource Lambda in MIE. Something in the spirit of this:

const resource = new cdk.CustomResource(this, `${operatorName}Operation`, {
  serviceToken,
  properties: {
    ResourceType: "Operation",
    Name: operatorName,
    Type: type,
    Configuration: configuration,
    StartLambdaArn: this.startLambda.handler.functionArn,
  },
});

const cfnResource = resource.node.defaultChild as cdk.CfnResource;
cfnResource.overrideLogicalId(`${operatorName}Operation`);

Jack Liddiard @deadcore 00:47

So during some further digging yesterday we discovered yesterday that the names are being truncated by what I assume is CDK - and removing the key word OperatorLibrary

Shorting the prefix to something like j1 was one work around but this does go against the spirit of our project - each developer has their own stack instead of sharing the same MIE insights.

Are you aware of any other possible work arounds/fixes/hacks? Or are we looking at bitting the bullet and sharing a MIE stack?