aws-sam-cli
aws-sam-cli copied to clipboard
Published
20 hours ago •
aws
aws
Bug: sam validate fails for template using 1.122.0 or 1.223.0
Description:
We have a template which validates and delays perfect using SAM CLI 1.121.0. But after upgrading it fails to validate with the error:
[[E0001: Error found when transforming the template] (Error transforming template: Key None must be a string) matched 1]
Error: Linting failed. At least one linting rule was matched to the provided template.
See template at the bottom!
I was able to 'debug' the template and the error seams to be a connector definition using type and are instead of an id.
eg:
Connectors:
AiWorkflowStabilisationFunctionS3Connector:
Properties:
Destination:
Type: AWS::S3::Bucket
Arn: !Sub arn:aws:s3:::qundo-legitimation-data-${Environment}
Permissions:
- Read
- Write
Steps to reproduce:
sam validate --lint --template-file template.yaml --region eu-central-1
Observed result:
sam validate --template-file /Users/dzwicker/projects/QUNDO/ai/ai-workflows/workflows/photo/template.yaml --region eu-central-1 --debug --lint
2024-08-22 18:26:22,220 | No config file found in this directory.
2024-08-22 18:26:22,222 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/Users/dzwicker/projects/QUNDO/ai/ai-workflows/workflows/photo/samconfig.toml'
2024-08-22 18:26:22,223 | Config file location: /Users/dzwicker/projects/QUNDO/ai/ai-workflows/workflows/photo/samconfig.toml
2024-08-22 18:26:22,223 | Config file '/Users/dzwicker/projects/QUNDO/ai/ai-workflows/workflows/photo/samconfig.toml' does not exist
2024-08-22 18:26:22,238 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/Users/dzwicker/projects/QUNDO/ai/ai-workflows/workflows/photo/samconfig.toml'
2024-08-22 18:26:22,238 | Using config file: samconfig.toml, config environment: default
2024-08-22 18:26:22,239 | Expand command line arguments to:
2024-08-22 18:26:22,239 | --template_file=/Users/dzwicker/projects/QUNDO/ai/ai-workflows/workflows/photo/template.yaml --lint
2024-08-22 18:26:22 Looking for CFLINTRC before attempting to load
2024-08-22 18:26:22 Validating User CFNLINTRC
2024-08-22 18:26:22 Validating CFNLINTRC config with given JSONSchema
2024-08-22 18:26:22 Schema used: {'$id': 'https://github.com/aws-cloudformation/cfn-python-lint/blob/main/src/cfnlint/data/CfnLintCli/config/schema.json', '$schema': 'http://json-schema.org/draft-07/schema#', 'additionalProperties': False, 'description': 'CFNLINTRC configuration schema', 'properties': {'append_rules': {'description': 'Location of directories to append rules from', 'items': {'type': 'string'}, 'type': 'array'}, 'configure_rules': {'additionalProperties': False, 'description': 'Configure rules', 'patternProperties': {'^.*$': {'patternProperties': {'^.*$': {'anyOf': [{'type': 'string'}, {'type': 'integer'}, {'type': 'boolean'}, {'items': {'type': 'string'}, 'type': 'array'}, {'items': {'type': 'integer'}, 'type': 'array'}, {'items': {'type': 'string'}, 'type': 'boolean'}]}}, 'type': 'object'}}, 'type': 'object'}, 'custom_rules': {'description': 'custom rule file to use', 'type': 'string'}, 'ignore_checks': {'description': 'List of checks to ignore', 'items': {'type': 'string'}, 'type': 'array'}, 'ignore_templates': {'description': 'Templates to ignore', 'items': {'type': 'string'}, 'type': 'array'}, 'include_checks': {'description': 'List of checks to include', 'items': {'type': 'string'}, 'type': 'array'}, 'mandatory_checks': {'description': 'List of mandatory checks to enforce', 'items': {'type': 'string'}, 'type': 'array'}, 'merge_configs': {'description': 'Merges lists between configuration layers', 'type': 'boolean'}, 'output_file': {'description': 'Path to the file to write the main output to', 'type': 'string'}, 'override_spec': {'description': 'Path to spec file to override with', 'type': 'string'}, 'regions': {'description': 'Regions to test against', 'items': {'type': 'string'}, 'type': 'array'}, 'registry_schemas': {'description': 'One or more directories of CloudFormation Registry Resource Schemas', 'items': {'type': 'string'}, 'type': 'array'}, 'templates': {'description': 'Templates to lint', 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'CFNLINTRC JSON Schema', 'type': 'object'}
2024-08-22 18:26:22 Config used: {}
2024-08-22 18:26:22 CFNLINTRC looks valid!
2024-08-22 18:26:22 Validating Project CFNLINTRC
2024-08-22 18:26:22 Validating CFNLINTRC config with given JSONSchema
2024-08-22 18:26:22 Schema used: {'$id': 'https://github.com/aws-cloudformation/cfn-python-lint/blob/main/src/cfnlint/data/CfnLintCli/config/schema.json', '$schema': 'http://json-schema.org/draft-07/schema#', 'additionalProperties': False, 'description': 'CFNLINTRC configuration schema', 'properties': {'append_rules': {'description': 'Location of directories to append rules from', 'items': {'type': 'string'}, 'type': 'array'}, 'configure_rules': {'additionalProperties': False, 'description': 'Configure rules', 'patternProperties': {'^.*$': {'patternProperties': {'^.*$': {'anyOf': [{'type': 'string'}, {'type': 'integer'}, {'type': 'boolean'}, {'items': {'type': 'string'}, 'type': 'array'}, {'items': {'type': 'integer'}, 'type': 'array'}, {'items': {'type': 'string'}, 'type': 'boolean'}]}}, 'type': 'object'}}, 'type': 'object'}, 'custom_rules': {'description': 'custom rule file to use', 'type': 'string'}, 'ignore_checks': {'description': 'List of checks to ignore', 'items': {'type': 'string'}, 'type': 'array'}, 'ignore_templates': {'description': 'Templates to ignore', 'items': {'type': 'string'}, 'type': 'array'}, 'include_checks': {'description': 'List of checks to include', 'items': {'type': 'string'}, 'type': 'array'}, 'mandatory_checks': {'description': 'List of mandatory checks to enforce', 'items': {'type': 'string'}, 'type': 'array'}, 'merge_configs': {'description': 'Merges lists between configuration layers', 'type': 'boolean'}, 'output_file': {'description': 'Path to the file to write the main output to', 'type': 'string'}, 'override_spec': {'description': 'Path to spec file to override with', 'type': 'string'}, 'regions': {'description': 'Regions to test against', 'items': {'type': 'string'}, 'type': 'array'}, 'registry_schemas': {'description': 'One or more directories of CloudFormation Registry Resource Schemas', 'items': {'type': 'string'}, 'type': 'array'}, 'templates': {'description': 'Templates to lint', 'items': {'type': 'string'}, 'type': 'array'}}, 'title': 'CFNLINTRC JSON Schema', 'type': 'object'}
2024-08-22 18:26:22 Config used: {}
2024-08-22 18:26:22 CFNLINTRC looks valid!
2024-08-22 18:26:22 User configuration loaded as
2024-08-22 18:26:22 {}
2024-08-22 18:26:22 Project configuration loaded as
2024-08-22 18:26:22 {}
2024-08-22 18:26:22 Merging configurations...
2024-08-22 18:26:22 Run scan of template None
2024-08-22 18:26:22 SAM Translator: 1.91.0
2024-08-22 18:26:22 Setting AWS_DEFAULT_REGION to eu-central-1
2024-08-22 18:26:22 Error transforming template: Key None must be a string
2024-08-22 18:26:22 Stack trace: Key None must be a string
Traceback (most recent call last):
File "/opt/homebrew/Cellar/aws-sam-cli/1.123.0/libexec/lib/python3.12/site-packages/cfnlint/template/transforms/_sam.py", line 156, in transform_template
sam_translator.translate(
File "/opt/homebrew/Cellar/aws-sam-cli/1.123.0/libexec/lib/python3.12/site-packages/samtranslator/translator/translator.py", line 188, in translate
translated = macro.to_cloudformation(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.123.0/libexec/lib/python3.12/site-packages/samtranslator/metrics/method_decorator.py", line 117, in wrapper_cw_timer
exec_result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.123.0/libexec/lib/python3.12/site-packages/samtranslator/model/sam_resources.py", line 1909, in to_cloudformation
self._add_connector_metadata(generated_resources, original_template, source, destination)
File "/opt/homebrew/Cellar/aws-sam-cli/1.123.0/libexec/lib/python3.12/site-packages/samtranslator/model/sam_resources.py", line 2183, in _add_connector_metadata
original_dest_type = original_resources.get(destination.logical_id, {}).get("Type")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/aws-sam-cli/1.123.0/libexec/lib/python3.12/site-packages/cfnlint/decode/node.py", line 77, in get
raise ValueError(f"Key {key!r} must be a string")
ValueError: Key None must be a string
[[E0001: Error found when transforming the template] (Error transforming template: Key None must be a string) matched 1]
2024-08-22 18:26:22,730 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-08-22 18:26:22,731 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2024-08-22 18:26:22,731 | Unable to find Click Context for getting session_id.
Error: Linting failed. At least one linting rule was matched to the provided template.
Expected result:
Should be validated as before.
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
template.yaml
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
ai-workflows-photo
Parameters:
Environment:
Type: String
AllowedValues:
- dev
- staging
- prod
- dz
LambdaRuntime:
Type: String
Default: provided.al2023
AllowedValues:
- provided.al2023
- java21
LambdaHandler:
Type: String
Default: not.used.in.provided.runtime
AllowedValues:
- not.used.in.provided.runtime
- io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler::handleRequest
LambdaArchitecture:
Type: String
Default: arm64
AllowedValues:
- arm64
- x86_64
SnsTopicSubscription:
Type: String
Default: "[email protected]"
Globals:
Function:
Runtime: !Sub "${LambdaRuntime}"
Architectures:
- !Sub "${LambdaArchitecture}"
Timeout: 20
# 128 is ok but to be fast
MemorySize: 128
Handler: !Sub "${LambdaHandler}"
Resources:
#Source SQS
SourceQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub ai-workflows-photo-queue-${Environment}
#Result SQS
ResultQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub ai-workflows-photo-result-queue-${Environment}
AiWorkflowStabilisationFunctionLog:
Type: AWS::Logs::LogGroup
Properties:
RetentionInDays: 7
LogGroupName: !Sub "/aws/lambda/ai-workflows-stabilisation-function-${Environment}"
#Lambda Stabilisation
AiWorkflowStabilisationFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub ai-workflows-stabilisation-function-${Environment}
CodeUri: ../../lambdas/stabilisation/build/function.zip
MemorySize: 1536
Policies:
- RekognitionLabelsPolicy: { }
Metadata:
SkipBuild: True
# noinspection YamlFormatViolation
Connectors:
AiWorkflowStabilisationFunctionS3Connector:
Properties:
Destination:
Type: AWS::S3::Bucket
Arn: !Sub arn:aws:s3:::qundo-legitimation-data-${Environment}
Permissions:
- Read
- Write
AiWorkflowStabilisationFunctionS3ConnectorDev:
Properties:
Destination:
Type: AWS::S3::Bucket
Arn: arn:aws:s3:::qundo-legitimation-data-dev
Permissions:
- Read
- Write
AiWorkflowStabilisationFunctionS3ConnectorTest:
Properties:
Destination:
Type: AWS::S3::Bucket
Arn: arn:aws:s3:::qundo-kyc-test-data
Permissions:
- Read
- Write
AiWorkflowExtractionFunctionLog:
Type: AWS::Logs::LogGroup
Properties:
RetentionInDays: 7
LogGroupName: !Sub "/aws/lambda/ai-workflows-extraction-function-${Environment}"
#Lambda Extraction
AiWorkflowExtractionFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub ai-workflows-extraction-function-${Environment}
CodeUri: ../../lambdas/extraction/build/function.zip
MemorySize: 192
Policies:
- TextractDetectAnalyzePolicy: { }
- TextractGetResultPolicy: { }
Metadata:
SkipBuild: True
# noinspection YamlFormatViolation
Connectors:
AiWorkflowExtractionFunctionS3Connector:
Properties:
Destination:
Type: AWS::S3::Bucket
Arn: !Sub arn:aws:s3:::qundo-legitimation-data-${Environment}
Permissions:
- Read
AiWorkflowExtractionFunctionS3ConnectorDev:
Properties:
Destination:
Type: AWS::S3::Bucket
Arn: arn:aws:s3:::qundo-legitimation-data-dev
Permissions:
- Read
AiWorkflowExtractionFunctionS3ConnectorTest:
Properties:
Destination:
Type: AWS::S3::Bucket
Arn: arn:aws:s3:::qundo-kyc-test-data
Permissions:
- Read
AiWorkflowExtractionMergeFunctionLog:
Type: AWS::Logs::LogGroup
Properties:
RetentionInDays: 7
LogGroupName: !Sub "/aws/lambda/ai-workflows-extraction-merge-function-${Environment}"
#Lambda Merge
AiWorkflowExtractionMergeFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub ai-workflows-extraction-merge-function-${Environment}
CodeUri: ../../lambdas/extraction-merge/build/function.zip
Metadata:
SkipBuild: True
#Logs for StepFunctions
AiWorkflowStateMachineLogGroup:
Type: AWS::Logs::LogGroup
Properties:
RetentionInDays: 7
LogGroupName: !Sub "/aws/snf/ai-workflows-${Environment}"
AiWorkflowFaceComparisonFunctionLog:
Type: AWS::Logs::LogGroup
Properties:
RetentionInDays: 7
LogGroupName: !Sub "/aws/lambda/ai-workflows-face-comparison-function-${Environment}"
#Lambda Face Comparision
AiWorkflowFaceComparisonFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: !Sub ai-workflows-face-comparison-function-${Environment}
CodeUri: ../../lambdas/face-comparison/build/function.zip
Policies:
- RekognitionFacesPolicy: { }
Metadata:
SkipBuild: True
# noinspection YamlFormatViolation
Connectors:
AiWorkflowFaceComparisonFunctionS3Connector:
Properties:
Destination:
Type: AWS::S3::Bucket
Arn: !Sub arn:aws:s3:::qundo-legitimation-data-${Environment}
Permissions:
- Read
AiWorkflowFaceComparisonFunctionS3ConnectorDev:
Properties:
Destination:
Type: AWS::S3::Bucket
Arn: arn:aws:s3:::qundo-legitimation-data-dev
Permissions:
- Read
AiWorkflowFaceComparisonFunctionS3ConnectorTest:
Properties:
Destination:
Type: AWS::S3::Bucket
Arn: arn:aws:s3:::qundo-kyc-test-data
Permissions:
- Read
AiWorkflowSnsTopic:
Type: AWS::SNS::Topic
Properties:
TopicName: !Sub ai-workflows-error-${Environment}
Subscription:
- Endpoint: !Ref SnsTopicSubscription
Protocol: "EMAIL"
#Target StepFunction state machine
AiWorkflowStateMachine:
Type: AWS::Serverless::StateMachine
Properties:
Name: !Sub ai-workflow-photo-${Environment}
DefinitionUri: workflow/ai-workflow-photo.asl.json
DefinitionSubstitutions:
AiWorkflowStabilisationFunctionArn: !GetAtt AiWorkflowStabilisationFunction.Arn
AiWorkflowExtractionFunctionArn: !GetAtt AiWorkflowExtractionFunction.Arn
AiWorkflowFaceComparisonFunctionArn: !GetAtt AiWorkflowFaceComparisonFunction.Arn
AiWorkflowExtractionMergeFunctionArn: !GetAtt AiWorkflowExtractionMergeFunction.Arn
AiWorkflowSnsTopicArn: !GetAtt AiWorkflowSnsTopic.TopicArn
Logging:
Destinations:
- CloudWatchLogsLogGroup:
LogGroupArn: !GetAtt AiWorkflowStateMachineLogGroup.Arn
Level: ALL
IncludeExecutionData: true
Policies:
- CloudWatchLogsFullAccess
- Version: '2012-10-17' # Policy Document
Statement:
- Effect: Allow
Action:
- geo:SearchPlaceIndexForText
Resource: 'arn:aws:geo:eu-central-1:001160930910:place-index/Qundo-Addresss-Check'
# noinspection YamlFormatViolation
Connectors:
AiWorkflowStepFunctionStabilisationLambdaWriteConnector:
Properties:
Destination:
Id: AiWorkflowStabilisationFunction
Permissions:
- Write
AiWorkflowStepFunctionExtractionLambdaWriteConnector:
Properties:
Destination:
Id: AiWorkflowExtractionFunction
Permissions:
- Write
AiWorkflowStepFunctionFaceComparisonLambdaWriteConnector:
Properties:
Destination:
Id: AiWorkflowFaceComparisonFunction
Permissions:
- Write
AiWorkflowExtractionMergeLambdaWriteConnector:
Properties:
Destination:
Id: AiWorkflowExtractionMergeFunction
Permissions:
- Write
AiWorkflowStepFunctionResultQueueWriteConnector:
Properties:
Destination:
Id: ResultQueue
Permissions:
- Write
AiWorkflowStepFunctionResultQueueWriteConnectorDev:
Properties:
Destination:
Type: AWS::SQS::Queue
Arn: arn:aws:sqs:eu-central-1:001160930910:ai-workflows-photo-queue-dev
Permissions:
- Write
AiWorkflowStepFunctionResultQueueWriteConnectorTest:
Properties:
Destination:
Type: AWS::SQS::Queue
Arn: arn:aws:sqs:eu-central-1:001160930910:ai-workflow-photo-result-for-test-*
Permissions:
- Write
AiWorkflowStepFunctionSnsTopicWriteConnector:
Properties:
Destination:
Id: AiWorkflowSnsTopic
Permissions:
- Write
# Logs for EventBridge Pipe
AiWorkflowSqsEventBridgePipeLogGroup:
Type: AWS::Logs::LogGroup
Properties:
RetentionInDays: 7
LogGroupName: !Sub "/aws/pipe/ai-workflows-${Environment}"
# Role for EventBridge Pipes to read from SQS and launch SFN
AiWorkflowSqsEventBridgePipeRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ai-workflow-sqs-event-bridge-pipe-role-${Environment}
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- pipes.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: CloudWatchLogs
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: '*'
- PolicyName: ReadSQS
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'sqs:ReceiveMessage'
- 'sqs:DeleteMessage'
- 'sqs:GetQueueAttributes'
Resource: !GetAtt SourceQueue.Arn
- PolicyName: ExecuteSFN
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'states:StartExecution'
Resource: !Ref AiWorkflowStateMachine
# EventBridge from SQS to StepFunction
AiWorkflowSqsEventBridgePipe:
Type: AWS::Pipes::Pipe
Properties:
Name: !Sub ai-workflow-photo-event-bridge-${Environment}
RoleArn: !GetAtt AiWorkflowSqsEventBridgePipeRole.Arn
DesiredState: RUNNING
LogConfiguration:
CloudwatchLogsLogDestination:
LogGroupArn: !GetAtt AiWorkflowSqsEventBridgePipeLogGroup.Arn
IncludeExecutionData:
- ALL
Level: TRACE
Source: !GetAtt SourceQueue.Arn
SourceParameters:
SqsQueueParameters:
BatchSize: 1
Target: !Ref AiWorkflowStateMachine
TargetParameters:
StepFunctionStateMachineParameters:
InvocationType: FIRE_AND_FORGET
InputTemplate: |
{
"body": <$.body>
}
Outputs:
AiWorkflowSourceQueueUrl:
Description: SourceQueue URL
Value: !GetAtt SourceQueue.QueueUrl
AiWorkflowResultQueueUrl:
Description: ResultQueue URL
Value: !GetAtt ResultQueue.QueueUrl
AiWorkflowStateMachineLogGroup:
Description: 'StepFunctions LogGroup Name'
Value: !Ref AiWorkflowStateMachineLogGroup
AiWorkflowStabilisationFunction:
Description: "Stabilisation Lambda Function ARN"
Value: !GetAtt AiWorkflowStabilisationFunction.Arn
AiWorkflowExtractionFunction:
Description: "Extraction Lambda Function ARN"
Value: !GetAtt AiWorkflowExtractionFunction.Arn
