aws-pdk
aws-pdk copied to clipboard
aws
[BUG] (type-safe-api) PrepareSpecCustomResource fail with error 413 in large API with many integrations
Describe the bug
When deploying an API using type-safe-api with a really large integrations object the PrepareSpecCustomResource fail with 413 because of large payload in input
Error:
1115107 byte payload is too large for the Event invocation type (limit 262144 bytes) (Service: AWSLambda; Status Code: 413; Error Code: RequestEntityTooLargeException; Request ID: 9f16fddd-3500-4106-aba5-07d6398ba67d; Proxy: null)
Expected Behavior
The PrepareSpecCustomResource process integrations successfully and deploy continue
Current Behavior
The PrepareSpecCustomResource fail and stop the deploy
Reproduction Steps
In order to reproduce the error it's pretty simple you need to make a really large integrations object and deploy this api
Here is the way I'm facing this issue in a real production scenario:
-
Follow the documentation in order to create the base
type-safe-apiproject in TS -
In the construct
myapi.tsadd the following constant and functions in order to create huge integration template responses
const RESPONSE_TEMPLATE = `#set($code = $input.body.split('<Code>')[1].split('</Code>')[0])
#set($message = $input.body.split('<Message>')[1].split('</Message>')[0])
#if($code == 'AccessControlListNotSupported')#set($errorCode = 'STOR1')
#elseif($code == 'AccessDenied')#set($errorCode = 'STOR2')
#elseif($code == 'AccessPointAlreadyOwnedByYou')#set($errorCode = 'STOR3')
#elseif($code == 'AccountProblem')#set($errorCode = 'STOR4')
#elseif($code == 'AllAccessDisabled')#set($errorCode = 'STOR5')
#elseif($code == 'AmbiguousGrantByEmailAddress')#set($errorCode = 'STOR6')
#elseif($code == 'AuthorizationHeaderMalformed')#set($errorCode = 'STOR7')
#elseif($code == 'BadDigest')#set($errorCode = 'STOR8')
#elseif($code == 'BucketAlreadyExists')#set($errorCode = 'STOR9')
#elseif($code == 'BucketAlreadyOwnedByYou')#set($errorCode = 'STOR10')
#elseif($code == 'BucketNotEmpty')#set($errorCode = 'STOR11')
#elseif($code == 'ClientTokenConflict')#set($errorCode = 'STOR12')
#elseif($code == 'CredentialsNotSupported')#set($errorCode = 'STOR13')
#elseif($code == 'CrossLocationLoggingProhibited')#set($errorCode = 'STOR14')
#elseif($code == 'EntityTooSmall')#set($errorCode = 'STOR15')
#elseif($code == 'EntityTooLarge')#set($errorCode = 'STOR16')
#elseif($code == 'ExpiredToken')#set($errorCode = 'STOR17')
#elseif($code == 'IllegalLocationConstraintException')#set($errorCode = 'STOR18')
#elseif($code == 'IllegalVersioningConfigurationException')#set($errorCode = 'STOR19')
#elseif($code == 'IncompleteBody')#set($errorCode = 'STOR20')
#elseif($code == 'IncorrectNumberOfFilesInPostRequest')#set($errorCode = 'STOR21')
#elseif($code == 'InlineDataTooLarge')#set($errorCode = 'STOR22')
#elseif($code == 'InternalError')#set($errorCode = 'STOR23')
#elseif($code == 'InvalidAccessKeyId')#set($errorCode = 'STOR24')
#elseif($code == 'InvalidAccessPoint')#set($errorCode = 'STOR25')
#elseif($code == 'InvalidAccessPointAliasError')#set($errorCode = 'STOR26')
#elseif($code == 'InvalidAddressingHeader')#set($errorCode = 'STOR27')
#elseif($code == 'InvalidArgument')#set($errorCode = 'STOR28')
#elseif($code == 'InvalidBucketAclWithObjectOwnership')#set($errorCode = 'STOR29')
#elseif($code == 'InvalidBucketName')#set($errorCode = 'STOR30')
#elseif($code == 'InvalidBucketState')#set($errorCode = 'STOR31')
#elseif($code == 'InvalidDigest')#set($errorCode = 'STOR32')
#elseif($code == 'InvalidEncryptionAlgorithmError')#set($errorCode = 'STOR33')
#elseif($code == 'InvalidLocationConstraint')#set($errorCode = 'STOR34')
#elseif($code == 'InvalidObjectState')#set($errorCode = 'STOR35')
#elseif($code == 'InvalidPart')#set($errorCode = 'STOR36')
#elseif($code == 'InvalidPartOrder')#set($errorCode = 'STOR37')
#elseif($code == 'InvalidPayer')#set($errorCode = 'STOR38')
#elseif($code == 'InvalidPolicyDocument')#set($errorCode = 'STOR39')
#elseif($code == 'InvalidRange')#set($errorCode = 'STOR40')
#elseif($code == 'InvalidRequest')#set($errorCode = 'STOR41')
#elseif($code == 'InvalidSecurity')#set($errorCode = 'STOR42')
#elseif($code == 'InvalidSOAPRequest')#set($errorCode = 'STOR43')
#elseif($code == 'InvalidStorageClass')#set($errorCode = 'STOR44')
#elseif($code == 'InvalidTargetBucketForLogging')#set($errorCode = 'STOR45')
#elseif($code == 'InvalidToken')#set($errorCode = 'STOR46')
#elseif($code == 'InvalidURI')#set($errorCode = 'STOR47')
#elseif($code == 'KeyTooLongError')#set($errorCode = 'STOR48')
#elseif($code == 'MalformedACLError')#set($errorCode = 'STOR49')
#elseif($code == 'MalformedPOSTRequest')#set($errorCode = 'STOR50')
#elseif($code == 'MalformedXML')#set($errorCode = 'STOR51')
#elseif($code == 'MaxMessageLengthExceeded')#set($errorCode = 'STOR52')
#elseif($code == 'MaxPostPreDataLengthExceededError')#set($errorCode = 'STOR53')
#elseif($code == 'MetadataTooLarge')#set($errorCode = 'STOR54')
#elseif($code == 'MethodNotAllowed')#set($errorCode = 'STOR55')
#elseif($code == 'MissingAttachment')#set($errorCode = 'STOR56')
#elseif($code == 'MissingContentLength')#set($errorCode = 'STOR57')
#elseif($code == 'MissingRequestBodyError')#set($errorCode = 'STOR58')
#elseif($code == 'MissingSecurityElement')#set($errorCode = 'STOR59')
#elseif($code == 'MissingSecurityHeader')#set($errorCode = 'STOR60')
#elseif($code == 'NoLoggingStatusForKey')#set($errorCode = 'STOR61')
#elseif($code == 'NoSuchBucket')#set($errorCode = 'STOR62')
#elseif($code == 'NoSuchBucketPolicy')#set($errorCode = 'STOR63')
#elseif($code == 'NoSuchCORSConfiguration')#set($errorCode = 'STOR64')
#elseif($code == 'NoSuchKey')#set($errorCode = 'STOR65')
#elseif($code == 'NoSuchLifecycleConfiguration')#set($errorCode = 'STOR66')
#elseif($code == 'NoSuchMultiRegionAccessPoint')#set($errorCode = 'STOR67')
#elseif($code == 'NoSuchWebsiteConfiguration')#set($errorCode = 'STOR68')
#elseif($code == 'NoSuchTagSet')#set($errorCode = 'STOR69')
#elseif($code == 'NoSuchUpload')#set($errorCode = 'STOR70')
#elseif($code == 'NoSuchVersion')#set($errorCode = 'STOR71')
#elseif($code == 'NotImplemented')#set($errorCode = 'STOR72')
#elseif($code == 'NotModified')#set($errorCode = 'STOR73')
#elseif($code == 'NotSignedUp')#set($errorCode = 'STOR74')
#elseif($code == 'OwnershipControlsNotFoundError')#set($errorCode = 'STOR75')
#elseif($code == 'OperationAborted')#set($errorCode = 'STOR76')
#elseif($code == 'PermanentRedirect')#set($errorCode = 'STOR77')
#elseif($code == 'PreconditionFailed')#set($errorCode = 'STOR78')
#elseif($code == 'Redirect')#set($errorCode = 'STOR79')
#elseif($code == 'RequestHeaderSectionTooLarge')#set($errorCode = 'STOR80')
#elseif($code == 'RequestIsNotMultiPartContent')#set($errorCode = 'STOR81')
#elseif($code == 'RequestTimeout')#set($errorCode = 'STOR82')
#elseif($code == 'RequestTimeTooSkewed')#set($errorCode = 'STOR83')
#elseif($code == 'RequestTorrentOfBucketError')#set($errorCode = 'STOR84')
#elseif($code == 'RestoreAlreadyInProgress')#set($errorCode = 'STOR85')
#elseif($code == 'ServerSideEncryptionConfigurationNotFoundError')#set($errorCode = 'STOR86')
#elseif($code == 'ServiceUnavailable')#set($errorCode = 'STOR87')
#elseif($code == 'SignatureDoesNotMatch')#set($errorCode = 'STOR88')
#elseif($code == 'SlowDown')#set($errorCode = 'STOR89')
#elseif($code == '503 SlowDown')#set($errorCode = 'STOR90')
#elseif($code == 'TemporaryRedirect')#set($errorCode = 'STOR91')
#elseif($code == 'TokenRefreshRequired')#set($errorCode = 'STOR92')
#elseif($code == 'TooManyAccessPoints')#set($errorCode = 'STOR93')
#elseif($code == 'TooManyBuckets')#set($errorCode = 'STOR94')
#elseif($code == 'TooManyMultiRegionAccessPointregionsError')#set($errorCode = 'STOR95')
#elseif($code == 'TooManyMultiRegionAccessPoints')#set($errorCode = 'STOR96')
#elseif($code == 'UnexpectedContent')#set($errorCode = 'STOR97')
#elseif($code == 'UnresolvableGrantByEmailAddress')#set($errorCode = 'STOR98')
#elseif($code == 'UserKeyMustBeSpecified')#set($errorCode = 'STOR99')
#elseif($code == 'NoSuchAccessPoint')#set($errorCode = 'STOR100')
#elseif($code == 'InvalidTag')#set($errorCode = 'STOR101')
#elseif($code == 'MalformedPolicy')#set($errorCode = 'STOR102')
#{else}#set($errorCode = 'STORX')#end
{"error": {"type": "storage_error","code": "$errorCode", "message": "$util.escapeJavaScript($message).replaceAll("\\'","'")"}, "meta": {}}`
function errorResponseSet(
responseTemplate: string,
): typesafeapi.IntegrationResponseSet {
return typesafeapi.IntegrationResponseSets.custom({
responses: {
...Object.fromEntries(
Array.of('400', '403', '404', '500').map((statusCode) => [
statusCode,
{
statusCode: statusCode,
responseParameters: {},
responseTemplates: {
'application/json': responseTemplate,
},
},
]),
),
'(4|5)\\d{2}': {
statusCode: '500',
responseParameters: {},
responseTemplates: {
'application/json': responseTemplate,
},
},
},
})
}
function cacheResponseSet(
responseTemplate: string,
): typesafeapi.IntegrationResponseSet {
return typesafeapi.IntegrationResponseSets.composite(
typesafeapi.IntegrationResponseSets.defaultPassthrough(),
errorResponseSet(responseTemplate),
)
}
- Then in
constructorcreate a CfnMapping in order to keep the Cloudformation Template as small as possible (this will allow to have a virtually big integrations object while keeping the cloudformation template low because it will use FindInMap)
const defaultMapping = new CfnMapping(scope, 'DefaultMapping', {
mapping: {
default: {
responseTemplate: RESPONSE_TEMPLATE,
},
},
})
- Then in
super.integrationscreate a lot ofintegrationwith similar props (you'll need to create in the OpenAPI spec a lot of fake route in order to do that)
getA: {
integration: typesafeapi.Integrations.s3({
bucket: props.cacheBucket,
method: 'get',
path: 'a.json',
integrationResponseSet: cacheResponseSet(
defaultMapping.findInMap('default', 'responseTemplate'),
),
}),
authorizer: typesafeapi.Authorizers.none(),
},
...
getZ: {
integration: typesafeapi.Integrations.s3({
bucket: props.cacheBucket,
method: 'get',
path: 'z.json',
integrationResponseSet: cacheResponseSet(
defaultMapping.findInMap('default', 'responseTemplate'),
),
}),
authorizer: typesafeapi.Authorizers.none(),
},
Possible Solution
In the PrepareSpecCustomResource there are several properties:
{
inputSpecLocation: {
bucket: inputSpecAsset.bucket.bucketName,
key: inputSpecAsset.s3ObjectKey,
},
outputSpecLocation: {
bucket: inputSpecAsset.bucket.bucketName,
key: preparedSpecOutputKeyPrefix,
},
...prepareSpecOptions,
};
and in prepareSpecOptions integrations and everything else about the api is passed directly, so in a context of a large integrations object, the process fail everytime.
A workaround could be to write all of the info into a bucket and then read the bucket.
Additional Information/Context
No response
PDK version used
v0.23.37
What languages are you seeing this issue on?
Typescript
Environment details (OS name and version, etc.)
macOS Sonoma 14.4.1 Apple M1 Pro
Hi @valebedu - thanks for raising this. I agree with your proposal - would you have capacity to raise a backwards compat PR to resolve this or would you be looking for us to add this for you? The only reason I bring this up is that we won't be able to realistically add this into the AWS PDK for another couple of weeks.
This issue is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon. If you wish to exclude this issue from being marked as stale, add the "backlog" label.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Closing this issue as it hasn't seen activity for a while. Please add a comment @mentioning a maintainer to reopen. If you wish to exclude this issue from being marked as stale, add the "backlog" label.
Reopening this as it's still an issue. Think it still needs a bit more digging/experimentation to figure out how we can handle large APIs without hitting the CFN/lambda size limits.
