Solve 4K Lambda Environment Variable Limit

[cheruvian]

Environment information

node ➜ /workspaces/amplify (main) $ npx ampx info
System:
  OS: Linux 6.10 Debian GNU/Linux 11 (bullseye) 11 (bullseye)
  CPU: (10) arm64 unknown
  Memory: 7.70 GB / 15.60 GB
  Shell: /bin/bash
Binaries:
  Node: 20.18.1 - /usr/local/bin/node
  Yarn: 1.22.22 - /usr/local/bin/yarn
  npm: 10.8.2 - /usr/local/bin/npm
  pnpm: 9.15.0 - /usr/local/share/npm-global/bin/pnpm
NPM Packages:
  @aws-amplify/auth-construct: 1.8.1
  @aws-amplify/backend: 1.16.1
  @aws-amplify/backend-ai: Not Found
  @aws-amplify/backend-auth: 1.7.1
  @aws-amplify/backend-cli: 1.7.2
  @aws-amplify/backend-data: 1.6.1
  @aws-amplify/backend-deployer: 2.1.1
  @aws-amplify/backend-function: 1.14.1
  @aws-amplify/backend-output-schemas: 1.6.0
  @aws-amplify/backend-output-storage: 1.3.1
  @aws-amplify/backend-secret: 1.4.0
  @aws-amplify/backend-storage: 1.4.1
  @aws-amplify/cli-core: 2.1.1
  @aws-amplify/client-config: 1.7.0
  @aws-amplify/data-construct: 1.16.1
  @aws-amplify/data-schema: 1.21.0
  @aws-amplify/deployed-backend-client: 1.7.0
  @aws-amplify/form-generator: 1.2.1
  @aws-amplify/model-generator: 1.2.0
  @aws-amplify/platform-core: 1.9.0
  @aws-amplify/plugin-types: 1.10.1
  @aws-amplify/sandbox: 2.1.2
  @aws-amplify/schema-generator: 1.4.0
  @aws-cdk/toolkit-lib: 0.3.2
  aws-amplify: 6.14.4
  aws-cdk-lib: 2.195.0
  typescript: 5.8.3
AWS environment variables:
  AWS_PROFILE = beta
  AWS_REGION = us-east-1
No CDK environment variables

Describe the feature

tl;dr; Amplify SSM Env config > 4KB Lambda env limit. 1+2 account for 50% of 4KB limit. 3 decouples the 2.

After using more than a handful (20 for us, 7 of which are added by amplify) environment variables/secrets, we hit the Lambda Environment variable limit. This allowed us an impractically limited 4 client secrets (2 API Keys + 2 OAuth Credentials)

[CloudFormationDeploymentError] The CloudFormation deployment has failed.
438
                                 ∟ Caused by: [_ToolkitError] The stack named amplify-XXXXXXXX-XXXXXX-branch-edda2b76ee failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE: 1 validation error detected: Value 'Lambda was unable to configure your environment variables because the environment variables you have provided exceeded the 4KB limit. String measured: {"XXXXXXXXXXXXXXX.....

The largest (for us > 2KB) environment variable is the AMPLIFY_SSM_ENV_CONFIG which is amplify's runtime lookup for ssm parameters.

There are a couple things that could help with this issue I suggest, 1 should be done with haste to mitigate the impact while 3 seems like a proper long term solution. 2 would also be impactful but potentially requires additional discussion. All of these solutions can be implemented in an opt-in fashion although I contend they should be done automatically for at least the production environment.

1. Remove unused variable injection

Amplify generates and injects "<value will be resolved during runtime>" as the value for every variable that it will inject, in our testing this contributed an additional 50% of unused environment variable bloat. These values are non functional and removal would provide customers some more runway until a better fix can be implemented. Removal may cause slight increase in debug challenges but the difference between a undefined value and "<value will be resolved during runtime>" is minimal, either way you know its an Amplify issue and in practice we bang our heads regardless when we see env var resolution issues.

Prior to adding 1 additional OAuth Credentials, we had:

• 3.7KB - Environment Variables
  • 1.9KB - AMPLIFY_SSM_ENV_CONFIG
  • 0.8KB - Amplify injected "<value will be resolved during runtime>" variables
  • 0.8KB - Actual (customer) environment variables.

Recommendation: Don't inject unused values into lambda environment vars.

2. Minified Serialization for (Amplify) names/identifiers

Focusing on just the AMPLIFY_SSM_ENV_CONFIG (1.9KB) we find the following

• 0.4KB - JSON/Keys overhead
• 0.25KB - User environment var names (not including SSM keys)
• 1.3KB - SSM Paths
  • 937B - Amplify repeated prefixes (e.g. /amplify/APP_ID/BRANCH_IDENTIFIER/)
  • 193B - Amplify var names (e.g. AMPLIFY_DATA_GRAPHQL_ENDPOINT, AMPLIFY_DATA_MODEL_INTROSPECTION_SCHEMA_BUCKET_NAME AMPLIFY_DATA_MODEL_INTROSPECTION_SCHEMA_KEY)
  • 175B - User var names

With this in mind, ~50% of the overhead can be addressed by serializing the App and Branch identifiers rather than repeating them N+6x. Another 10% can be reclaimed with shorter amplify var names, and another 10% if we consider something besides JSON, although the cost benefit tradeoff for those 2 are limited.

Recommendation: Move to resolving SSM prefix at runtime and serializing App and Branch identifier once.

3. Hardcoded/code injected AMPLIFY_SSM_ENV_CONFIG

Amplify already does code injection into the lambda function to load/refresh the SSM parameters. There's actually no need to pass this in via environment variable at all. Instead I see 3 potential options:

1. Inject the env config directly as JSON
2. Store the env config in SSM at a hardcoded path

Injecting the env config directly into the code as JSON seems to be viable since the paths are generated in the FunctionEnvironmentTranslator and don't include any CDK tokens. It could be concatenated into/before the injected banner and will then be used to refresh the SSM params. The downside is that this can't be updated dynamically, however that doesn't seem to be the use case anyways and updating environment variables requires a container restart anyways.

Storing env config in SSM at a hardcoded (function) path would also allow decoupling of the SSM config from the lambda env vars. However it requires 2 serial calls to SSM (first get config, second get params) which would increase e2e latency. This could be an opt in feature or trigger if the env vars are seen to surpass the 4KB limit.

Recommendation: Inject env config directly into code.

Use case

Support increased amounts of environment variables to allow > 4 client/API integrations.